Paper: pdf
Slides: pdf
Citation:
APA
Halfaker, A., Gieger, R. S., Morgan, J., & Riedl, J. (2013). The Rise and Decline of an Open Collaboration System: How Wikipedia's reaction to sudden popularity is causing its decline. American Behavioral Scientist 57(5) 664-688, DOI:10.1177/0002764212469365.
bibtex
@article{halfaker13rise, author={Aaron Halfaker and R. Stuart Geiger and Jonathan Morgan and John Riedl}, title={The Rise and Decline of an Open Collaboration System: How Wikipedia's reaction to sudden popularity is causing its decline}, year={2013}, month={May}, volume={57}, number={5}, pages={664--688}, journal={American Behavioral Scientist}, doi={10.1177/0002764212469365}, url={http://dx.doi.org/10.1177/0002764212469365} }

The Rise and Decline of an Open Collaboration Community: How Wikipedia's reaction to sudden popularity is causing its decline

Open collaboration systems like Wikipedia need to maintain a pool of volunteer contributors in order to remain relevant. Wikipedia was created through a tremendous number of contributions by millions of contributors. However, recent research has shown that the number of active contributors in Wikipedia has been declining steadily for years, and suggests that a sharp decline in the retention of newcomers is the cause. This paper presents data that show that several changes the Wikipedia community made to manage quality and consistency in the face of a massive growth in participation have ironically crippled the very growth they were designed to manage. Specifically, the restrictiveness of the encyclopedia's primary quality control mechanism and the algorithmic tools used to reject contributions are implicated as key causes of decreased newcomer retention. Further, the community's formal mechanisms for norm articulation are shown to have calcified against changes – especially changes proposed by newer editors.

Authors

Key Findings (tl:dr)

To deal with the massive influx of new editors between 2004 and 2007, Wikipedians built automated quality control tools and solidified their rules of governance. These reasonable and effective strategies for maintaining the quality of the encyclopedia have come at the cost of decreased retention of desirable newcomers.

  1. The decline represents a change in the rate of retention of desirable, good-faith newcomers.
  2. Semi-autonomous vandal fighting tools (like Huggle) are partially at fault.
  3. New users are being pushed out of policy articulation.

Summary

Gentle reader,

Below is the authors' summary of a paper that was accepted to a special issue of American Behavioral Scientist on Wikis. I also provide the unabridged, pre-print of the paper if you desire more discussion and explanation.

Feel free to email me with questions, and please let me know if you find an error.

Enjoy!
Aaron Halfaker (aaron.halfaker@gmail.com)

According to a report published in 2009 by the Wikimedia Foundation, the number of active editors working on the English Wikipedia is declining. As the figure 1 below suggests, the number of active editors (editors with >= 5 edits/month) abruptly stopped growing in early 2007 and entered a steady, linear decline. Recent research has shown evidence that this transition is rooted in the declining retention of new editors, not a change in the retention of already-experienced old-timers (Suh, 2009). What is unclear, or was before this work, is why this sudden change in the retention of new editors took place.

Figure 1. The editor decline. The number of active editors (>=5 edits/month) is plotted over time for the English language Wikipedia.

This paper implicates the strategies adopted by Wikipedia editors to preserve the quality and consistency of the encyclopedia in causing the decline in retention of desirable newcomers. Below, the results are broken up into a description of three general findings.

The decline in desirable newcomers

One of the biggest open questions about Wikipedia's newcomer decline was whether it was the result of a natural decline in the quality of newcomers (where lower-quality newcomers were "encouraged" to go elsewhere) or whether changes in how Wikipedia welcomes newcomers was at fault. In order to explore this, we manually categorized the work of 2100 newcomers sampled over the history of the website. With the help of Maryana Pinchuk (Accedie), Oliver Keyes (Ironholds) and Steven Walling (Steven_Walling) we categorized these newcomers into 4 ordinal quality classes based on their first session of editing activity:

  1. Vandals - Purposefully malicious, out to cause harm
  2. Bad-faith - Trying to be funny, not here to help or harm
  3. Good-faith - Trying to be productive, but failing
  4. Golden - Successfully contributing productiv

In the analysis, we simplify these 4 categories into ''desirable'' newcomers (good-faith & golden) and ''undesirable'' newcomers (bad-faith & vandal).

Figure 2. Proportion of desirable newcomers. The proportion of newcomers falling into the two desirable quality classes, good-faith and golden, is plotted over time.

Figure 3. Rejection of desirable newcomers. The proportion of desirable newcomers who are reverted in their first session of editing is plotted over time.

Figure 4. Survival of desirable newcomers. The proportion of desirable newcomers who continue to make edits for at least two months is plotted over time.

The three plots above (and the logistic regression models we built) make a few things apparent:

Efficient quality control lead to an impersonal newcomer experience

Figure 5. First message to new users. The proportion of first messages to new editors is plotted over time by the way in which those messages were delivered.

In order to maintain the quality of encyclopedic content in the face of exponential growth in the contributor community, Wikipedians developed automated (bots) and semi-automated tools (Huggle, Twinkle, etc.) to make the work of rejecting undesirable contributions waste as little effort as possible. These tools are apparently effective at their job. Recent research has shown that the time between when vandalism is posted and reverted is very short (median: ~2 minutes)(Kittur, 2007) and has been steadily falling(West, 2010). However, we hypothesized that the ways in which efficiency was achieved with these tools was part of the problem.

Recent research by Geiger et al. has shown that an increasing amount of newcomers' first message received after joining Wikipedia is business end of an automated quality control tool (Geiger, 2012). Figure 5 shows the growing use of automated tools to send messages to newcomers. Figure 6 shows that the use of these quality control tools to revert newcomers is growing and our regression models (see the paper) suggest that desirable newcomers who were reverted with them were especially less likely to continue editing.

Figure 6. Automated reverts of desirable newcomers. The proportion newcomers reverted by semi-automated tools.

Figure 7. BRD reciprocation rate. The rate at which reverting editors respond to BRD initiations is plotted over time by the tool used to perform the revert.

In order to explain this effect, we looked for evidence that bot users were interacting negatively to newcomers, buy examining their adherence to Wikipedia's best practices for discussions about reverted edits, the Bold, revert, discuss cycle. As figure 7 shows, Huggle users were susprisingly unlikely to respond to desirable newcomers when they questioned why they were reverted.

Calcification of rules against newcomers

Recent work studying Wikipedia's policies and guidelines has suggested that the process by which these rules and recommendations are vetted reflect community concerns and decentralization in governance participation (Beschastnikh, 2008). However, it's also been shown that experienced Wikipedians have more power over interpretation of the rules (Kriplean, 2007) and that policy and guideline creation has slowed since 2006 (Forte, 2009) We suspected that, although natural and generally beneficial, that this calcification of the rules of Wikipedia would biased against newcomers' concerns.

To explore this hypothesis, we built of regression model predicting which edits to policies and guidelines were likely to be reverted. In short, this model showed us that:

Figure 8. Growth in the rules of governance. The growth rate of policies, guidelines and essays is plotted over time.

While we do not suggest that the rules of Wikipedia be open to reinterpretation by newcomers, we advocate that concern should be allocated for newcomers with a legitimate interest in changing the way that Wikipedia works. We argue that these results suggest how important it is that newcomers have a say in how they are treated.

References

The views and opinions expressed in this page are strictly those of the page author.
The contents of this page have not been reviewed or approved by the University of Minnesota.