Jump to content

英文维基 | 中文维基 | 日文维基 | 草榴社区

User:Tzusheng/sandbox/Validation Study

From Wikipedia, the free encyclopedia

The validation study of Wikibench.

Background

[edit]

This validation study is part of the broader research that aims to support Wikipdians to evaluate AI systems, such as ORES and LiftWing, by curating an evaluation dataset that Wikipdians can use to compare against AI's predictions.

To support the curation of the dataset in the context of edit quality, we developed Wikibench, a wiki plug-in that enables Wikipedians to label whether an edit is damaging or saved with good or bad faith while patrolling recent changes. Wikibench also has several features that facilitate discussion when Wikipedians disagree on the label. During our pilot testing, Wikibench received overwhelmingly positive feedback from Wikipedians.

The purpose of this validation study

[edit]

The last mile of this pilot testing is to validate whether the data and its label curated using Wikibench indeed reflect community consensus. Therefore, as shown in the table below, we sample a set of edits from Wikibench's dataset and invite experienced Wikipedians to collectively discuss and decide what the "correct" label of these edits should be. We'll compare the labels Wikipedians enter in this table to Wikibench's labels to understand how many of them are identical or different and to know the effectiveness of Wikibench in terms of "correctness".

How to participate

[edit]

For invited Wikipedians, please enter your label in the column of your username. For edit damage, the label is either damaging or not damaging. For user intent, the label is either good faith or bad faith.

After labeling all the edits, please be bold to enter or update the consensus based on your and others' labels in the consensus column. Whenever there are disagreements, please discuss them on the talk page. Note that you don't necessarily have to wait for others' labels to enter the consensus.

The study completes when the consensus columns are filled with labels that are reflective of the consensus of the five participating experienced Wikipedians. Note that consensus is not necessarily the majority label.

Study duration

[edit]

This validation study lasts about a week, from July 14 to 22, 2023. It might be extended for another week if Wikipedians find the extension helpful for building consensus.

Compensation

[edit]

We'd provide $90 USD to compensate for your time, experience, and contribution. As discussed in the village pump and concluded here, this research is not the paid editing that Wikipedians and WMF are concerned with. However, you may still consider adding the following userbox for disclosure if you decide to accept our compensation.

{{User:Tzusheng/Userboxes/WikibenchDisclose}}

Per Terms of Use, this user discloses being paid by CMU as part of a study for contributing to Wikibench.




Selected edits for validation

[edit]
Selected edits for the validation study
Diffs Edit damage User intent
Title Diff IDs Zippybonzo Skarmory Bluerasberry Matthewrb Schminnte Consensus Zippybonzo Skarmory Bluerasberry Matthewrb Schminnte Consensus
Adjacent 645892572/677829222 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith good faith
Elephant goad 667271580/677974402 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith good faith
Indian name 680080662/680081690 not damaging not damaging 💙 damaging Not damaging not damaging Not damaging good faith good faith 💙 good faith Good faith good faith good faith
Empire (2015 TV series) 681052757/681086400 damaging not damaging 💙 damaging Not damaging not damaging not damaging good faith good faith 💙 good faith Good faith good faith good faith
List of awards and nominations received by Carly Rae Jepsen 682416765/682418557 damaging damaging 💙 damaging Damaging damaging Damaging good faith good faith 💙 good faith Good faith good faith Good faith
T. A. Shahid 664715336/683209659 damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Ollie Hayes 680497781/683511584 damaging not damaging 💙 damaging Not damaging not damaging not damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Panathinaikos B.C. 684796093/684882507 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Mr Green (company) 677934765/685548844 Not damaging damaging 💙 not damaging Damaging damaging not damaging Good faith good faith 💙 bad faith Good faith good faith Good faith
Montreal Canadiens 691789091/691911798 damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Tales of Mystery and Imagination (Nocturnal Rites album) 618199239/692384164 Damaging damaging 💙 damaging Damaging damaging Damaging Bad faith good faith 💙 bad faith Good faith bad faith bad faith
Ilocano language 694270988/694271323 Damaging damaging 💙 damaging Damaging damaging Damaging Bad faith bad faith 💙 bad faith Bad faith bad faith Bad faith
Masters Champions League 695181325/696058558 Damaging not damaging 💙 damaging Damaging damaging damaging Bad faith good faith 💙 bad faith Bad faith bad faith bad faith
List of Telugu films of 2015 698373777/698392149 Damaging not damaging 💙 damaging Damaging damaging damaging Good faith good faith 💙 good faith Good faith good faith good faith
Betty Holberton 691777400/699667798 Not damaging not damaging 💙 damaging Not damaging not damaging not damaging Good faith good faith 💙 bad faith Good Faith good faith good faith
User talk:Bishnu Rasaily prev/703976291 Not damaging not damaging 💙 not damaging Not damaging not damaging Not damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Soldier Fuel 704809356/705203406 Not damaging not damaging 💙 not damaging Not damaging not damaging Not damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Ousmane Dembélé 705814739/705815021 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith bad faith 💙 good faith Good faith bad faith bad faith
Heather O'Rourke 705812016/706027789 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith good faith
List of presidents of the United States by age 708655010/708695027 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith bad faith 💙 good faith Bad faith bad faith bad faith
Charlotte Douglas International Airport 713303042/713770261 Not damaging not damaging 💙 damaging Not damaging not damaging Not damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Melt 710265936/714932920 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Sam Kim 715352163/715353867 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith Good faith
2016 Fort McMurray wildfire 719225065/719236236 Damaging not damaging 💙 damaging Damaging damaging damaging Good faith good faith 💙 good faith Bad faith good faith Good faith
St. Mark's School of Texas 719347634/719359416 Not damaging not damaging 💙 not damaging Not damaging not damaging Not damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Joi Lansing 715632333/719925637 Damaging damaging 💙 damaging Damaging damaging Damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Robert Wadlow 720771351/720797296 Not damaging not damaging 💙 not damaging Not damaging not damaging Not damaging Good faith good faith 💙 good faith Good faith good faith Good faith
N.O.R.E. 722177677/722563742 Not damaging not damaging 💙 not damaging Not damaging not damaging not damaging Good faith good faith 💙 good faith Good faith good faith Good faith
Nelson Story 719484034/723407464 not damaging damaging 💙 not damaging Damaging damaging damaging good faith good faith 💙 good faith Good faith good faith Good faith
Talk:Bollywood films and plagiarism 670938590/726418154 not damaging not damaging 💙 not damaging Not damaging not damaging Not damaging good faith good faith 💙 good faith Good faith good faith Good faith
2016 Novak Djokovic tennis season 724757924/726787424 not damaging not damaging 💙 damaging Not Damaging not damaging Not damaging good faith good faith 💙 bad faith Good faith good faith good faith
Witold Sobociński 726553825/730265683 not damaging not damaging 💙 not damaging Not damaging not damaging Not damaging good faith good faith 💙 good faith Good faith good faith Good faith
The Adventures of Super Mario Bros. 3 723218057/731155214 damaging damaging 💙 damaging Damaging damaging Damaging bad faith bad faith 💙 bad faith Bad faith bad faith bad faith
State Road 10 (Serbia) 735558460/735558653 not damaging not damaging 💙 not damaging Not damaging not damaging Not damaging good faith good faith 💙 good faith Good faith good faith Good faith