VisualizIR: a game for identifying and categorizing relevant text in documents

Authors:
Christopher G. Harris
Affiliations:
The University of Iowa, Iowa City, Iowa
Venue:
Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design
Year:
2012

Citing 3
Cited 0

Designing games with a purpose

Communications of the ACM - Designing games with a purpose
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Quality through flow and immersion: gamifying crowdsourced relevance assessments

SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper, we introduce VisualizIR, a game where players identify relevant document terms that match predefined categories. VisualizIR evaluates players on accuracy, recall, and precision against an established gold standard, a pooled consensus of judgments made by other players, or a weighted combination of the two. The annotated document can then viewed by any XML-compatible browser, allowing for quick identification of terms in the document related to each category. Here we describe some of the playability design tradeoffs made during the game's development, as well as our findings from two experiments conducted using VisualizIR output.