Wikipedia:Historical archive/Policy/Approval mechanism

This is not a Wikipedia policy and does not reflect current Wikipedia practice. It is a proposed set of policies for discussion. This page was refactored and moved to article validation. As of 2009, current proposals for approval mechanisms can be found at Wikipedia:Flagged revisions.

Wikipedia approval mechanism, defined

"Wikipedia approval mechanism" means any sort of mechanism whereby Wikipedia articles are individually marked and displayed, somehow, as "approved".

Purpose

The purpose of an approval mechanism is, essentially, quality assurance. By presenting particular articles as approved, we (Wikipedians) would be representing those articles as reliable sources of information.

Proposed basic requirements

Among the basic requirements of an approval mechanism would have to fulfill in order to be adequate are:

• The approval must be done by experts about the material approved. A trust model?
• There must be clear and reasonably stringent standards that the experts are expected to apply.
• The mechanism itself must be genuinely easy for the experts to use or follow. Nupedia's experience seems to show that a convoluted approval procedure, while it might be rigorous, is too slow to be of practical use.
• The approval mechanism must not impede the progress of Wikipedia in any way. It must not change the Wikipedia process; it should be an "add-on".
• Must not be a bear to program, and it shouldn't require extra software or rely on browser-specific stuff like Java (or Javascript) that some users won't have. A common browser platform standard must be specified for them, preferably low level but not so low level that the machines don't have CDs.
• Must provide some way of verifying the expert's credentials—and optionally a way to verify that he or she approved the article, not an imposter.

Some results we might want from an approval system are that it:

• makes it possible to broaden or narrow the selection of approvers (E.g., one person might only wish authors who have phd's, another would allow for anyone who has made an effort to approve any articles.),
• allows for extracting topic-oriented sets (e.g., in order to produce an "Encyclopedia of Music") (The idea is that article approval could contain more information than just the binary "high-quality" bit, e.g. topic area, level of detail, and so forth. Such "approved metadata" would allow easy extraction of user-defined subsets of the full approved article set.), and
• provides a framework for allowing multiple sets of "approvals", allow not just one set of approvers or one approval rating but different sets to allow for different standards. (I.e. as with consumer products, different safety approvals, scuba diving certifications, etc.)

Advantages

The advantages of an approval mechanism of the sort described are clear and numerous:

• We will encourage the creation of really good content.
• Large, reputable websites and the web in general are more likely to use and|or link to our content if it has been approved by experts. And, especially, if the current version of an article has a persistent URL that they can link to. At present only a past version of an article actually has such a URL!
• The addition of an approval mechanism will be attractive to academics who might not participate without it—particularly the academics who might want to be reviewers. All that matters is a mass of GNU FDL text on important topics that is good enough for serious scholars to find worth correcting. EofT
• What does "All that matters is a mass of GNU FDL text on important topics that is good enough for serious scholars to find worth correcting." mean? Brianjd 10:22, 2005 Jan 29 (UTC)
• It makes it easier to collect the best articles on Wikipedia and create completed "snapshots" of them that could be printed and distributed, for example. This issue is central to Pushing to 1.0.

Generally, Wikipedia will become comparable to nearly any encyclopedia, once enough articles are approved. It need not be perfect, just better than Britannica and Encarta and ODP and other CD or Web resources.

Disadvantages

I am not sure there are any significant disadvantages of an approval mechanism, but idly, I think there might be one. I think that it's possible that Wikipedia might become more of an "exclusive club" than it is, if people start comparing nascent articles contributed by new contributors to the finished projects. I might not want to contribute two sentences about widgets if I think ten neat paragraphs, with references, is what is expected. Again, I don't know if this is really apt to be a problem.

Another general argument against is that this really doesn't seem necessary. An approval mechanism has been suggested since Day One of Wikipedia, and evidence aside that Wikipedia is working just fine, will probably continue to be suggested 'til kingdom come.

• An expert-centered approval mechanism might be considered a hierarchic methodology, in contrast with the bazaar-type open-source projects like wikipedia, that are known to achieve good results (e.g. Linux) through aggressive peer-review, and openness ("With enough eyeballs, all errors are shallow"). It can be argued that the very reason Linux has become so reliable is the radical acceptance, and for some degree, respect, for amateurs' and enthusiasts' work of all sorts.

• Experts have controversies among themselves: for example, many subjects in medicine and psychology are highly debated. By giving a professor the free hand in deciding whether an article is "approved" or "non-approved" there is a risk of compromising the NPOV standards by experts' over-emphasizing their specific opinions and area of research.

• Finding an expert who corresponds to a certain article can sometimes be troublesome:
  • Can a Ph.D on applied Mathematics "approve" articles on pure mathematics?, or more strictly, should one be accepted as a approver only if he/she have made research on the specific subject he/she is approving? Who will decide whether a person is qualified for approval?
  • Some obscure or day-to-day topics don't have any immediate "expert" attached to them. Who will approve articles on hobbies, games, local cultures etc.?

Is it needed?

These are arguments presented for why an additional approval mechanism is unnecessary for wikipedia:

• Wikipedia already has an approval mechanism; although anyone can edit any page and experts and amateurs of all levels can be bold and contribute to articles, communal scrutiny is an approval mechanism, though not as centrally-controlled as peer review.

• Imperfect as it currently stands, Wikipedia already fosters a more sophisticated critical approach to all sources of information than simple reliance on "peer-reviewed" authority. Low-quality articles can be easily recognized by a reader with some or no experience in reading wikipedia, and by applying some basic critical thinking:
  • The style may sound biased, emotional, poorly written, or just unintelligible.
  • Blanket statements, no citing, speculative assertions: any critical person will be careful in giving too much credit for such article.
  • The history of an article shows much of the effort and review that has been brought into writing it, who and how qualified are the writers (Users seem to put some biographical information about themselves on their pages.)
  • a sophisticated reader soon learns that the Talk page is often enlightening as to the processes that have resulted in the current entry text.

• Cross-checking with other sources is an extremely important principle for good information gathering on the internet! No source should be taken as 100% reliable.

• Some "authoritative" and "approved" encyclopedias don't seem to stand for their own claims of credability. See, for example, Columbia Encyclopedia's article about Turing test, compared with Wikipedia's Turing test. "Human-like conversation" is not the same as "human-like thought". See also Meta:Errors in the Encyclopædia Britannica which have been corrected in Wikipedia.

• The very idea of an article being "approved" is debatable, especially on controversial topics, and can be seen as an unreachable ideal by some.

Proposals

Below, we can develop some specific proposals for approval mechanisms.

Sanger's proposal

When I say the approval mechanism must be really easy for people to use, I mean it. I mean it should be extremely easy to use. So what's the easiest-to-use mechanism that we can devise that nevertheless meets the criteria?

The following: on every page on the wiki, create a simple popup approval form that anyone may use. ("If you are a genuine expert on this subject, you can approve this article.") On this form, the would-be article approver [whom I'll call a "reviewer"] indicates name, affiliation, relevant degrees, web page (that we can use to check bona fides), and a text statement to the effect of what qualifications the person has to approve of an article. The person fills this out (with the information saved into their preferences) and hits the "approve" button.

When two different reviewers have approved an article, if they are not already official reviewers, the approval goes into moderation.

The approval goes into a moderation queue for the "approved articles" part of Wikipedia. From there, moderators can check over recently-approved articles. They can check that the reviewers actually are qualified (according to some pre-set criteria of qualification) and that they are who they say they are. (Perhaps moderator-viewable e-mail addresses will be used to check that a reviewer isn't impersonating someone.) A moderator can then "approve the approver".

The role of the moderators is not to approve the article, but to make sure that the system isn't being abused by underqualified reviewers. A certain reviewer might be marked as not in need of moderation; if two such reviewers were to approve of an article, the approval would not need to be moderated.

New addition I think it might be a very good idea to list, on an approved article, who the reviewers are who have approved the article.

--Larry Sanger

I think it would be a very good idea to list, on any article, the names of ANY reviewers to have reviewed the article. Seeing who disapproved it is, perhaps, more important than seeing who approved it, since Wikipedia generally has high quality articles anyway. Brianjd 10:22, 2005 Jan 29 (UTC)

Bryce's proposal

From my experience with the Wikipedia_NEWS, it seems that there's a lot that can be done with the wiki software as it exists. The revision control system and its tracking of IP addresses is ok as a simple screen against vandalism. The editing system seems fairly natural and is worth using for managing this; certainly we can expect anyone wishing to be a reviewer ought to have a fair degree of competence with it already.

Second, take note at how people have been making use of the user pages. People write information about themselves, the articles they've created, and even whole essays about opinions or ideas.

What I'd propose is that we encourage people who wish to be reviewers to set up a subpage under their userpage called '/Approved?'. Any page that they added to this page is considered to be acceptable by them. (It is recommended they list the particular revision # they're approving too, but it's up to them whether to include the number or not.) The reviewer is encouraged to provide as much background and contact information about themselves on their main page (or on a subpage such as /Credentials?) as they wish. It is *completely* an opt-in system, and does not impact wikipedia as a whole, nor any of its articles.

Okay, so far it probably sounds pretty useless because it *seems* like it gives zero _control_ over the editors. But if we've learned nothing else from our use of Wiki here, it's that sometimes there is significant power in anarchy. Consider that whoever is going to be putting together the set of approved articles (let's call her the Publisher) is going to be selecting the editors based on some criteria (only those with phds, or whatever). The publisher has (and should have) the control over which reviewers they accept, and can grab their /Approved? lists at the time they wish to publish. Using the contact info provided by the reviewer, they can do as much verification as they wish; those who provide insufficient contact info to do so can be ignored (or asked politely on their userpage.) But the publisher does *not* have the power to control whether or not you or I are *able* to approve articles. Maybe for the "PhD? Reviewers Only" encyclopedia I'd get ruled out, but perhaps someone else decides to do a "master's degree or better" one, and I would fit fine there. Or maybe someone asks only that reviewers provide a telephone number they can call to verify the approved list.

Consider a further twist on this scheme: In addition to /Approved?, people could set up other specific kinds of approval. For instance, some could create /Factchecked? pages where they've only verified any factual statements in the article against some other source; or a /Proofed? page that just lists pages that have been through the spellchecker and grammar proofer; or a /Nonplagiarised page that lists articles that the reviewer can vouch for as being original content and not merely copied from another encyclopedia. The reason I mention this approach is that I imagine there will be reviewers who specialize in checking certain aspects of articles, but not everything (a Russian professor of mathematics might vouch for everything except spelling and grammar, if he felt uncomfortable with his grasp of the English language). Other reviewers can fill in the gaps (the aformentioned professor could ask another to review those articles for spelling and grammar, and they could list them on their own area.

I think this system is very in keeping with wiki philosophy. It is anti-elitist, in the sense that no one can be told, "No, you're not good enough to review articles," yet still allows the publisher to discriminate what to accept based on the reviewer's credentials. It leverages existing wiki functionality and Wikipedia traditions rather than requiring new code and new skills. And it lends itself to programmatic extraction of content. It also puts a check/balance situation between publisher and reviewer: If the publisher is selecting reviewers to include unfairly, someone else can always set up a fairer approach. There is also a check against reviewer bias, because once discovered, ALL of their reviewed articles would be dropped by perhaps all publishers, which gives a strong incentive to the reviewer to demonstrate the quality of their reviewing process and policies.

-- BryceHarrington

We would need some standards for this to work. If the subpages are named differently and/or have different structures it will be too difficult to use.

Each publisher would have to check the credentials themselves. Leaving results of checks on user's pages/subpages is not acceptable; anyone can edit it; developers can edit the history. Brianjd 10:22, 2005 Jan 29 (UTC)

Magnus Manske's proposal

I'll try to approach the whole approval mechanism from a more practical perspective, based on some things that I use in the Wikipedia PHP script. So, to set up an approval mechanism, we need

• • Namespaces to separate different stages of articles,
  • User rights management to prevent trolls from editing approved articles.

From the Sanger proposal, the user hierarchy would have to be

1. 1. Sysops, just a handful to ensure things are running smoothly. They can do everything, grant and reject user rights, move and delete articles etc.,
   2. Moderators who can move approved articles to the "stable" namespace,
   3. Reviewers who can approve articles in the standard namespace (the one we're using right now),
   4. Users who do the actual work. ;)

Stages 1-3 should have all rights of the "lowerlevels", and should be able to "rise" other users to their level. For the namespaces, I was thinking of the following:

• • The blank namespace, of course, which is the one all current wikipedia articles are in; the normal wikipedia
  • An approval namespace. When an article from "blank" gets approved by the first reviewer, a copy goes to the "approval" namespace.
  • A moderated namespace. Within the "approval" namespace, no one can edit articles, but reviewers can either hit a "reject" or "approve" button. "Reject" deletes the article from the "approval" namespace, "approve" moves it to the "moderated" namespace.
  • A stable namespace. Same as for "approval", but only moderators can "reject" or "approve" an article in "moderated" namespace. If approved, it is moved to the "stable" namespace. End of story.

This system has several advantages:

• • By having reviewers and moderators not chosen for a single category (e.g., biology), but by someone on a "higher level" trusting the individual not to make strange decisions, we can avoid problems such as having to choose a category for each article and each person prior to approval, checking reviewers for special references etc.
  • Reviewers and moderators can have special pages that show just the articles currently in "their" namespace, making it easy to look for topics they are qualified to approve/reject
  • Easy handling. No pop-up forms, just two buttons, "approve" and "reject", throughout all levels.
  • No version confusion. The initial approval automatically locks that article in the "approval" namespace, and all decisions later on are on this version alone.
  • No bother of the normal wikipedia. "Approval" and "moderated" can be blanked out in every-day work, "stable" can be blanked out as an option.
  • Easy to code. Basically, I have all parts needed ready, a demo version could be up next week.

Ehrenberg addition

This would be added on to any of the above approval proceses. After an article is approved, it would go into the database of approved articles. People would be able to access this from the web. After reading an article, the reader would be able to click on a link to disapprove of the article. After 5 (more, less?) people have disapproved of an article, the article goes through a reapproval process, in which only one expert must approve it, and then the necessary applicable administrators. -- Suggested addition: there should be a separate domain, perhaps frozenwikipedia.org, which includes only approved articles. This could be used as a "reliable" reference when factual accuracy was very important.

This could be used as a "reliable" reference when factual accuracy was very important.

No, it couldn't. If factual accuracy is very important, you must check several sources, which means that it wouldn't make much difference whether you used the "approved" version or not. Brianjd 10:22, 2005 Jan 29 (UTC)

DWheeler's Proposal: Automated Heuristics

It might also be possible to use some automated heuristics to identify "good" articles. This could be especially useful if the Wikipedia is being extracted to some static storage (e.g., a CD-ROM or PDA memory stick). Some users might want this view as well. The heuristics may throw away some of the latest "good" changes, as long as they also throw away most of the likely "bad" changes.

Here are a few possible automated heuristics:

• Ignore all anonymous changes; if someone isn't willing to have their name included, then it may not be a good change. This can be "fixed" simply by a some non-anonymous person editing the article (even trivially).
• Ignore changes from users who have only submitted a few changes (e.g., less than 50). If a user has submitted a number of changes, and is still accepted (not banned), then the odds are higher that the user's changes are worthwhile.
• Ignore pages unless at least some number of other non-anonymous readers have read the article and/or viewed its diffs (e.g., at least 2 other readers). The notion here is that, if someone else read it, then at least some minimal level of peer review has occurred. The reader may not be able to identify subtle falsehoods, but at least "Tom Brokaw is cool" might get noticed. This approach can be foiled (e.g., by creating "bogus readers"), but many trolls won't bother to do that.

These heuristics can be combined with the expert rating systems discussed elsewhere here. An advantage of these automated approaches is that they can be applied immediately.

Other automated heuristics can be developed by developing "trust metrics" for people. Instead of trying to rank every article (or as a supplement to doing so), rank the people. After all, someone who does good work on one article is more likely to do good work on another article. You could use a scheme like Advogato's, where people identify how much they respect (trust) someone else. You then flow down the graph to find out how much each person should be trusted. For more information, see Advogato's trust metric information. Even if the Advogato metric isn't perfect, it does show how a few individuals could list other people they trust, and over time use that to derive global information. The Advogato code is available - it's GPLed.

Another related issue might be automated heuristics that try to identify likely trouble spots (new articles or likely troublesome diffs). A trivial approach might be to have a not-publicly-known list of words that, if they're present in the new article or diffs, suggest that the change is probably a bad one. Examples include swear words, and words that indicate POV (e.g., "Jew" may suggest anti-semitism). The change might be fine, but such a flag would at least alert someone else to especially take a look there.

A more sophisticated approach to automatically identify trouble spots might be to use learning techniques to identify what's probably garbage, using typical text filtering and anti-spam techniques such as naive Bayesian filtering (see Paul Graham's "A Plan for Spam"). To do this, the Wikipedia would need to store deleted articles and have a way to mark changes that were removed for cause (e.g., were egregiously POV) - presumably this would be a sysop privilege. Then the Wikipedia could train on "known bad" and "known good" (perhaps assuming that all Wikipedia articles before some date, or meeting some criteria listed above, are "good"). Then it could look for bad changes (either in the future, or simply examining the entire Wikipedia offline).

A trivial approach might be to have a not-publicly-known list of words that, if they're present in the new article or diffs, suggest that the change is probably a bad one.

Why does it have to be not-publicly-known? Brianjd 10:22, 2005 Jan 29 (UTC)

I assume the idea is that if the list was known, it would be quicker for vandals to think of alternative nasty things to say. But really we would want to be watching all the changes anyway, so that by watching bad edits that got past the filter one could update the list. But really I think this naughtiness flagging is over complicated and not very useful. More helpful might be a way to detect subtle errors: change in dates, heights, etc. that casual proofreaders would be less likely to pick up. Rather than simply flagging the edit, maybe highlighting the specific change in the normal version of the article to warn all readers: "Colombus discovered Antarctica? in 1492.". Reviewers would then be able to right click on the change and flag it as OK or bad, for example. Luke 05:34, 15 Feb 2005 (UTC)

PeterK's Proposal: Scoring

This idea has some of the same principles as the Automated Heuristic suggested above. I agree that an automated method for determining "good" articles for offline readers is absolutely crucial. I have a different idea on how to go about it. I think the principles of easy editing and how wikipedia works now is what makes it great. I think we need to take those principles along with some search engine ideas to give a confidence level for documents. So people extracting the data for offline purposes can decide the confidence level they want and only extract articles that meet that confidence level.

I think the exact equation for the final scoring needs to be discussed. I don't think I could come up with a final version by myself, but I'll give an example of what would give good point and bad points.

Final Score: a: first thing we need it a quality/scoring value for editors. Anonymous editors would be given a value of 1 and a logged in user may get 1 point added to their value for each article he/she edits, up to a value of 100. b: 0.25 points for each time a user reads the article c: 0.25 point for each day the article has existed in wikipedia d: each time the article is edited it gets 1+(a/10)*2 points, anonymous user would give it 1.2 and a fully qualified user would give it 20 points. e: next if an anonymous user makes a large change then you get a -20 point deduction. Even though this is harsh, if it goes untouched for 80 days it will gain all those points back. It will gain the points back faster if a lot of people have read the article.

This is the best I can think of right now, if I come up with a better scoring system I'll make some changes. Anyone feel free to test score a couple of articles to see how this algorithm holds up. We can even get a way of turning the score to a percentage, so that people can extract 90% qualified articles.

next if an anonymous user makes a large change then you get a -20 point deduction.

This one is very dangerous. What is the threshold for a "large" change? If we set it too high, this won't work very well (it will presumedly react to blanking a page, but I think this is most often used as a substitute for the "nominate for deletion" link that should be there). If we set it too low, and several edits are made by anonymous users to a not-very-popular article, then it could give a low (negative?) score, even though the article is of a high quality. Brianjd 10:22, 2005 Jan 29 (UTC)

Anyone feel free to test score a couple of articles to see how this algorithm holds up.

How? We don't know the scores for the editors. The only way to test this algorithm properly is to download some articles and the entire contributions lists of all the contributors! Brianjd 10:22, 2005 Jan 29 (UTC)

Legion of Trolls proposal

Trolls are not here to approve, and usually reject views of experts who must be certified by someone trolls grumble about. So one would expect them to be disgruntled by definition about such a mechanism. However, paradoxically, almost all trolls think they apply clear and reasonably stringent standards. The problem is that each troll has his own standards, unlike those of others!

That said, there is much to agree on: the mechanism itself must be genuinely easy to use, nothing slow and rigorous is of any value, the progress of Wikipedia and its proven process should not be impeded, and the results of the approval can be ignored. Where trolls would disagree is that verifying the expert's credentials are of any value. Any such mechanism can be exploited, as trolls know full well, often being experts at forging new identities and the deliberate disruption of any credentialing mechanism.

One might ignore this, and the trolls, but, it remains that what goes on at Wikipedia is largely a process not of approval but of impulse and then disapproval. As with morality and diplomacy, we move from systems of informal to formal disapproval. Today, even our reality game shows demonstrate the broad utility of this approach, with disapproval voting of uninteresting or unwanted or undesired candidates a well-understood paradigm.

So, imagine an entirely different way to achieve the "desirements", one that is a natural extension of Wikipedia's present process of attempt (stubs, slanted first passes, public domain documents, broad rewrites of external texts) and disapproval (reverts, neutralizing, link adds, rewrites, NPOV dispute and deletions). Rather than something new (trolls hate what is new) and unproven that will simply repeat all the mistakes of academia. Imagine a mechanism that

• Begins with all approved, and makes it possible to broaden or narrow the selection of approvers (e.g., one person might only wish authors who have phd's, another would allow for anyone who has made an effort to approve any articles) for each reader, or supported class of reader, simply by disapproving editors.
• Allows for extracting topic-oriented sets (e.g., in order to produce an "Encyclopedia of Music") relying on metadata that is specific to each such supported class of reader, not part of the Wikipedia as a whole
• Exploits ongoing feedback ("I don't care about this." or "I don't understand this.") to adjust the list of articles of interest. Each user can begin from some class (like Simple-English-only readers), and adjust if they like.
• Potentially, exploits more feedback on authors ("I can't believe this." or "I find this irrelevant.") to adjust also the list of disapproved authors/editors.
• Credits each troll who has driven off a disapproved author or editor. OK, that's a joke, but what do you expect, I'm a troll neh neh neh...

By embracing and extending and formalizing the disapproval, boredom and disdain that all naturally feel as a part of misanthropy, we can arrive at a pure and effective knowledge resource. One that rarely tells us what we don't care about. And, potentially, one that can let us avoid those who we find untruthful.

Propose and veto

Include articles that have been proposed by at least one person, and vetoed by none.

Where two versions of an article are so approved, pick the later one. Where no versions of an article are so approved, have no article.

That's it.

This is too easily abused. It adds too much load to the server for the dismal results it will produce. Brianjd 10:22, 2005 Jan 29 (UTC)

Andrew A's proposal

See m:Referees. This proposal is consistent with much of the above.

Main features:

• No impact on the existing way of doing things other than appearance of extra links on some article pages, saying that this article has been reviewed and linking to details.

• All articles remain in the main namespace. The only new namespace is a special one used to keep track of who has reviewed what.

• All reviews relate to a specific version. As any update to the article creates a new version, this initially has no reviewed status at all (with some possible exceptions where there is no need to maintain independence between contributor and reviewer and the software might therefore flag the approval by default). The concepts of freezing articles or promoting them in a configuration management system are not needed.

• All reviews are simply check boxes or radio buttons. Discussion remains on existing talk pages.

• QA on the main Wikipedia is supplied by three levels of reviewer:
  • A basic level which we initially give to anyone who asks for it,
  • A specialist level at which the reviewer claims expertise on the subject of the article,
  • A referee level at which the reviewer claims to be a quotable authority on the subject of the article.

• At the referee level only, the reviewer is expected to be independent of the article author(s). At other levels, reviewers are encouraged to edit and refactor as necessary, but not to review articles which are primarily their own work.

• At all levels, collegiate responsibility is the system, identical to the existing Wikipedia. There is no attempt to formally identify areas of expertise. Other members at this level are expected to comment if a particular reviewer is performing badly, and particularly if they are venturing outside their areas of expertise at the higher levels.

• At all levels, all reviewers are expected to approve only articles that they consider of high quality.

• All three levels allow for dissent from the approval of others.

• The same software (with extensions) can support
  • Wikipedia 1.0,
  • A G-rated Wikipedia,
  • Specialist encyclopedias.

• Readers of Wikipedia must be offered user-friendly ways of restricting what they see to their desired level of review and (hopefully) reliability. These might include
  • User options,
  • URL aliases (e.g. www.family.en-two.iwiki.icu, www.refereed.en-two.iwiki.icu).

David Levinson's proposal (revised Jan 2005)

Establishing Release Edition

The "Wikipedia Release Edition" would be a (hopefully large) subset of "Wikipedia Working Edition".

Users could browse the release edition or the working edition. When browsing, a header would note whether or not the latest version in the working edition was the same as the release edition.

At the bottom of every page of Wikipedia there would be two buttons: "Approve" and "Disapprove".

Any logged in user can vote once. Article versions would be scored based on number of votes for approval minus votes for disapproval. The article version with the highest score (so long as it is greater than 1) would be the released article.

A user who clicks "edit this page" would be presented with the newest version, and a note if this is not the Release Edition version. (showing differences)

Any save is automatically a vote in favor of that version and a removal of approval votes for every previous versions by that user.

Recent changes would note if a new version becomes the release version.

Advantages

• Doesn't rely on "expert model",
• Uses consensus approach, the article with the most favorable votes, and more favorable than unfavorable votes, is released;
• If votes go awry, edit wars begin, sensible users can vote to "disapprove" bringing score negative;
• Release edition can happen quickly, no approval queue waiting for votes, only one voter beside author is required (assuming the article garners no disapprovals).

Disadvantages

• An older version with lots of votes in favor may be difficult to dislodge without active observation.
• Editing may be more complicated with different versions for release and working edition.
• Votes can be manipulated by a user with multiple log-ins.

Establishing Quality Assurance

The release edition depends on votes. It also may lead to the perception that there is a fork, or create confusion about multiple wikipedias. A more "wiki" way of doing things would be to have only one release version, but allow anyone to rate any version. This system exploits version control to give users the ability to know who has "certified" a version.

So if I know that revision 01.22.2005.10:33 of an article has been certified by User:Authority, User:Hobbyist and User:Professional, I may trust it more than an article certified by User:Vandal or User:Random.

Users who want to influence what other people read will need to establish credibility. Real world information helps here, as well as within-Wikipedia credibility.

Given the potential number of Users who might certifiy an article, and the likelihood I have never heard of most of them, users should be allowed to join certification "clubs" (or rather "clubs" should be allowed to include members, through a process like Wikipedia:Requests_for_adminship). There might be a group who certifies articles on biology, and another group that certifies articles on astronomy, and so on. There could be multiple clubs who certify articles (biology and astronomy might both certify an article on exobiology. There might be more than one astronomy club, if there are differences, and good articles would be certified by both, and controversial articles by only one. Clubs could team, so if Club:Biology and Club:Astronomy trusted each other, they would be part of a Team (say Team:Science) (and of course there might be multiple teams certifying the same article, or different versions, which would ultimately need to be hashed out in a wiki way or would stand).

If club members went off the reservation so to speak, and certified rubbish, they could be kicked out of the club, and they would no longer be able to speak for the club (there certifications would no longer hold the club imprimatur). (similarly clubs could be kicked out of teams).

The important thing is that no one is requiring that only credentialed individuals be permitted to certify an article, but that if they do, users can of their own free will give that more credance than an uncredentialed person certifying the article.

Does this make wikipedia more complicated? Yes of course it does.

Does this make wikipedia more reliable? Yes, as you would now know who thought what was accurate.

Additions to MediaWiki required by this

• A way of tracking who certified each version,
• A way of tracking who was in what club (and what club was in what team).
• A certification history of articles.
• At the top of the page it would notify me if there was a more recent uncertified version of the article, or the most recent version of the article certified by Team:X or Club:Y.
• A certification or approval tab.
• The ability to view only articles approved by Team:X, Club:Y, or User:Z. (This could be done like categories, but there is probably a better mechanism)
• The ability to see Recent Approvals.

dml 17:09, 22 Jan 2005 (UTC)

dml

Giles Robertson's proposal

There doesn't need to be a central approval mechanism; approval is all about trust, and centrally dictating who can be trusted raises too many issues. That being said, an approval mechanism that relies on everybody voting is open to abuse, and doesn't carry the weight of approval by authority.

Instead, a decentralised proposal [which is similar to Andrew A's proposal], would be for authorities to create pages that list why they are an authority, and then list the pages and versions that they have approved. This could be done without any change to the existing codebase, but a system that marked some pages as "authority conferrers", and that marked on each page which authorities had approved it, would improve the usability.

Authority conferring pages could confer authority on to other authority-conferring pages; the aim is to build a 'web of trust' somewhat similar to that which provides the assurement in PGP keys that people are who they say they are. That said, it is important to prevent authority-conferring pages from fabrication; it may be necessary to lock these pages to certain users.

How do we establish that an authority-conferring page has authority? If we link it to another page, and that page displays text or graphic (with the Wikipedia logo) confirming that the external page trusts the authoritiy-conferring page, then we know, if the external page is free from interference, that the authority-conferrer is trusted. How strong that is as an approval, depends on what the external page(s) for the authority-conferrer are.

This also allows a selective extraction of approved pages: Pick an authority-conferrer, and extract every page that is approved by that authority, and, at your option, other pages at lower levels (e.g. page approved by authority trusted by authority you've just chosen).

The major disadvantage is that this introduces yet more metadata, in the form of the authority-conferrers. It also does not stop the creation of spurious authority-conferrers; though they won't be trusted or have much external approval, they may mislead. Any save is automatically a vote in favor of that version and a disapproval of every previous versions.

Recent changes would note if a new version becomes the release version.

Giles Robertson

m.e's proposal

I have a proposal at User talk:M.e/Production Wikipedia for creating 'production quality' pages. Highlight is the idea of collecting 'issues' and 'endorsements' against a 'frozen' version of a page. Hopefully we can then converge on a revised version that just fixes the issues. m.e. 12:16, 12 Sep 2004 (UTC)

Szopen's proposal

meta:Pending edits. Maybe not directly about approval mechanisms, but similar. Edits would be delayed for a fixed amount of time, and if none would object to them, they would be authomatically approved.

ChrisG proposal

main article: User:ChrisG/Approval_mechanism

This is a suggested approval mechanism to meet the need to make Wikipeda a more reliable source and to support Wikipedia 1.0. It is a twist on the namespace workflow mechanism suggested by Magnus Manske here. It is a process firmly based on wiki principles. It is flexible because decisions about quality are based on human judgment that each article meets the minimum standards for approval for Wikipedia 1.0. If it is agreed that standards should be raised then humans can amend their judgment appropriately during approval debates for more modern versions.

Aims

The design aims to:

• Have no effect on the wiki process for creating and developing articles.
• No fork within Wikipedia
• Identify specific versions of articles as being stable and hence give the reader the comfort that the flagged version has been reviewed as meeting certain minimum standards.
• Identify specific versions of articles as suitable for Wikipedia 1.0.
• The process needs to be scaleable. It needs to be able to approve a large percentage of Wikipedia within 3-6 months.
• Mark versions of each article appropriately depending on the outcome of any approval process for that version. This gives a guide to quality of particular versions in the history.
• Be simple to understand.
• Be easy to manage.
• Be democratic.
• Avoid requiring any exclusive, 'expert' clubs.
• Be relatively simple to code.
• Difficult to game.
• Evolutionary approach rather than fundamental change in Wikipedia process.
• Be acceptable to the majority of Wikipedians.

Overview of the process

The process makes use of namespaces to create a workflow process. A copy of the recommended version of an article is moved through the system, with an attached discussion page and votes. The process is best illustrated visually:

Maureen's proposal 1

The following was proposed by User:Maurreen on Wikipedia talk:Forum for Encyclopedic Standards 06:49, 19 Nov 2004 (UTC), but seems appropriate to mention here. -- Jmabel | Talk 22:45, Nov 23, 2004 (UTC)

<begin Maurreen's post>
Here's an outline of a possible plan, submitted for your suggestions. One advantage is that the computer part of it is simple. Maurreen 06:49, 19 Nov 2004 (UTC)

1. An article could be "approved" with at least 10 votes and no more than 10 percent of the votes objecting. Only registered users could vote. Users with multiple accounts should only vote once. Voting for each article would be open at least a week.
2. To allow appropriate time for review and possibly improvement, no more than three articles would be considered at a time. Others nominated would be compiled on a "pending" list.
3. Nominations would be accepted only from registered users who agree to maintain the article. So, in a sense, we would be voting on the nominators also. The commitment to maintain the article would allow the article to still be edited, but give us some assurance that the article wouldn't deteriorate.
4. "Approved" articles would be compiled on a list, along with the names of the maintainers.
5. Possibly the nominations or list could refer or be limited to a specific version of the article. That is, the article at such-and-so date and time.
6. "Approved" articles could have some indicator of that status on the article itself.
7. Guidelines or standards for what is worthy could be determined before voting ever starts on articles.
8. Nominations would be encouraged from featured articles and peer-reviewed articles. Initially, general topics (such as Electronics) would be preferred to more-specific ones (such as Ohm's Law). That could work toward the "approved" material having a broad and even general base.
9. Nominations of contentious articles would be discouraged, at least initally.

<end Maurreen's post>

Maurreen's proposal 2: "Reviewed articles"

Copied from Wikipedia talk:Forum for Encyclopedic Standards. Maurreen 14:12, 27 Nov 2004 (UTC)

Advantages:

• Simple and quick. This could be implemented and useful within an hour of its adoption.
• Takes best-of-both-worlds approach to wiki nature and any standing of experts.

Outline:

1. Any registered user could review any article.
2. There would be a category and list of reviewed articles.
3. The list would indicate which version of an article was reviewed by each reviewer.
4. Reviews on the list would be no more than a paragraph long.

Options:

1. Detailed reviews could be written and linked to from the list.
2. Reviewers who chose to could list themselves and a paragraph about any relevant qualifications or limitations on a list of reviewers.
3. Reviewers should at least indicate if they have worked on the article.
4. We could have a list of articles for which a review is desired, or use the current peer review page.
5. We could choose a set of suggested levels or other indicators (such as “acceptable,” “weak,” “comprehensive,” etc.).

Maurreen 21:04, 21 Nov 2004 (UTC)

---

Interim measure

Maybe it would be helpful to think of my proposal for article reviews as an interim measure. It isn't intended to be perfect by any means.

It is intended to give readers some measure of the quality of any given article or article version.

It is something that could very easily be produced and used while something better is discussed, decided and developed. It does not preclude any other system. It can include, or not include, a minimum standard for Wikipedia articles, which would need to be developed.

It could be one of any number of tools that work toward an eventual paper or "release" version. Maurreen 09:34, 25 Nov 2004 (UTC)

Editorial board(s)

Copied from Wikipedia talk:Forum for Encyclopedic Standards. Maurreen 14:12, 27 Nov 2004 (UTC)

My thought on this is that there is an immediate problem with 'Any registered user could review any article'. There's no problem with having anyone review an article - however to improve the credibility and reliability of Wikipedia, I think that we (unfortunately) need a link to the 'outside world' where people have real names and qualifications, rather than 'karma' built up under a nom-de-plume. For articles' credibility to be increased, someone's, or some people's, reputation needs to be on the line. My thoughts are that a properly constituted editorial board needs to approve (and possibly modify) articles. As I've mentioned elsewhere, there could (and in my view should) be multiple competing boards aiming to set their seal upon particular article versions. For example, one such board could be a set of academics in a particular subject whose names are known, who have a publishing record in peer-reviewed journals, and have an academic reputation. This does not preclude a self selected group of people setting up their own board under noms-de-plume and producing a Wikireputation based set of approvals. Users would have the choice of using either or both or neither board's seals of approval (article tags) as a filter into Wikipedia. WLD 21:48, 22 Nov 2004 (UTC)

ChrisG's template and process

Copied from Wikipedia talk:Forum for Encyclopedic Standards. Maurreen 14:12, 27 Nov 2004 (UTC)

Thinking some more about Maurreen's proposal it occurred to me that with the combination of templates and categories we could set up a voting system to approve articles. Consider this template (I used subst to create the text, e.g. {{subst:ChrisGtest}} ):

If you look to the bottom of the screen this template categorises the article as a candidate for Wikipedia 0.1 and also by the current day, month and year (uses variables so need to update). This means anybody wishing to vote on articles need only check the appropriate category for articles. They could click the links to the talk page of the articles they are interested in. The talk page of the article would give the link to the specific version and the votes so far; after checking the article the person could vote as they see fit.

As time passes, the articles listed as candidates will dwindle as they are approved or rejected. The fact the candidates are categorised by date would mean we know when to close the vote of any articles that have been sitting in candidate status for too long.

Rejected articles would have the candidate category removed. Successful articles would be given a Wikipedia 0.1 category instead, again identified by the date of the version approved. In addition the specific version of the article should be listed somewhere as approved on that date, i.e Wikipedia: Approved 0.1/1 Dec 2004.

I realize we don't have a consensus on how to approve articles, largely I think because some people are talking about approving top quality articles and others are talking about minimum standards for the CD/DVD editions; but this is a method which we could apply now without changes to the software, which would scale and would thus be suitable for either purpose.

ChrisG 01:10, 25 Nov 2004 (UTC)

Hawstom's proposal

If we want a mechanism that works right now to improve our credibility a little bit without sacrificing the principles we are all comfortable with, it must be simple and open. I propose that we simply start with the existing technical hierarchy of anon/user/admin/bureaucrat/developer to create a system that can assign a level of confidence to every version of every article, then display by default the most recent version with a disclaimer and links if the confidence level is low.

Principles

An article is only as trustworthy as the last hand that touched it. Wiki works because of immediate gratification.

Viewing

We continue to show to the public the latest version. If the confidence level for that article version is lower than our established standard, we show a disclaimer along with link to any more trusted article versions available. We put in user preferences a selection for the level of credibility to show by default(Developer/Bureaucrat/Admin/User/Anonymous), and we enable anon users, via a cookie or session id, to say, "Show only article versions with credibility level 2 or higher."

Editing

For users with high permissions, we put a check box or radio button on the editing page so that they may save with artificially low trust interim work with remaining unresolved credibility problems.

How it's simple

Nobody has to take the time to review articles. The approval occurs naturally as a by-product of the way we already work.

Since an article is only as trustworthy as the last hand that touched it and wiki works because of immediate gratification, only natural and open methods such as this can work at the Wikipedia.

How it improves our credibility

Places editorial responsibility on our trusted users. While it is true we have expressly disclaimed any editorial authority for our trusted users, the implicit attitude of the community has been to give them that authority. This proposal simply recognizes the de facto arrangement. In the future, new levels of editorial authority may be created.

Endorsement idea

Copied from Wikipedia talk:Forum for Encyclopedic Standards. Maurreen 14:12, 27 Nov 2004 (UTC)

We might be able to develop a system by which anyone could endorse a particular version of an article; presumably groups could form whose endorsement would carry some weight. The mechanism would be one "reader" approval equals point one(0.1) an "editor" would rank one point (1) an "editing librarian" would rank one to ten (1 - 10) an admin would rank 10 to eleven (10 - 11), a group could be assigned a similarly weighted scoring rank, i.e. an opinion offered editorially by "The Royal College of Physicians & Surgeons" might rank one way or the other compared to a select group of its Alumni. Deriving from this an articles approval-rating would be a function of its veracity as regards the opinion of the majority of its' readers. an entry in wikipedia would have (available for review) an articles position relevant to all other articles. (Idea from anon, Maurreen moved from project page.)