Quality control mechanisms for crowdsourcing: peer review, arbitration, & expertise at familysearch indexing

  • Authors:

  • Derek L. Hansen;Patrick J. Schone;Douglas Corey;Matthew Reid;Jake Gehring

  • Affiliations:

  • Brigham Young University, Provo, Utah, USA;FamilySearch, Salt Lake City, Utah, USA;Brigham Young University, Provo, Utah, USA;Brigham Young University, Provo, Utah, USA;FamilySearch, Salt Lake City, Utah, USA

  • Venue:
  • Proceedings of the 2013 conference on Computer supported cooperative work
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

The FamilySearch Indexing project has enabled hundreds of thousands of volunteers to transcribe billions of records, making it one of the largest crowdsourcing initiatives in the world. Assuring high quality transcriptions (i.e., indexes) with a reasonable amount of volunteer effort is essential to keep pace with the mounds of newly digitized documents. Using historical data, we show the relationship between prior experience and native language on transcriber agreement. We then present a field experiment comparing the effectiveness (accuracy) and efficiency (time) of two quality control mechanisms: (1) Arbitration -- the existing mechanism wherein two volunteers independently transcribe records and disagreements go to an arbitrator, and (2) Peer Review -- a mechanism wherein one volunteer's work is reviewed by another volunteer. Peer Review is significantly more efficient, though not as effective for certain fields as Arbitration. Design suggestions for FamilySearch Indexing and related crowdsourcing initiatives are provided.