Eliciting Informative Feedback: The Peer-Prediction Method

  • Authors:
  • Nolan Miller;Paul Resnick;Richard Zeckhauser

  • Affiliations:
  • Kennedy School of Government, Harvard University, Cambridge, Massachusetts 02138;School of Information, University of Michigan, Ann Arbor, Michigan 48109-1092;Kennedy School of Government, Harvard University, Cambridge, Massachusetts 02138

  • Venue:
  • Management Science
  • Year:
  • 2005

Quantified Score

Hi-index 0.01

Visualization

Abstract

Many recommendation and decision processes depend on eliciting evaluations of opportunities, products, and vendors. A scoring system is devised that induces honest reporting of feedback. Each rater merely reports a signal, and the system applies proper scoring rules to the implied posterior beliefs about another rater's report. Honest reporting proves to be a Nash equilibrium. The scoring schemes can be scaled to induce appropriate effort by raters and can be extended to handle sequential interaction and continuous signals. We also address a number of practical implementation issues that arise in settings such as academic reviewing and online recommender and reputation systems.