Seeding simulated queries with user-study data for personal search evaluation

  • Authors:
  • David Elsweiler;David E. Losada;José C. Toucedo;Ronald T. Fernandez

  • Affiliations:
  • University of Erlangen, Erlangen, Germany;Universidad de Santiago de Compostela, Santiago de Compostela, Spain;Universidad de Santiago de Compostela, Santiago de Compostela, Spain;Universidad de Santiago de Compostela, Santiago de Compostela, Spain

  • Venue:
  • Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper we perform a lab-based user study (n=21) of email re-finding behaviour, examining how the characteristics of submitted queries change in different situations. A number of logistic regression models are developed on the query data to explore the relationship between user- and contextual- variables and query characteristics including length, field submitted to and use of named entities. We reveal several interesting trends and use the findings to seed a simulated evaluation of various retrieval models. Not only is this an enhancement of existing evaluation methods for Personal Search, but the results show that different models are more effective in different situations, which has implications both for the design of email search tools and for the way algorithms for Personal Search are evaluated.