Differences in search engine evaluations between query owners and non-owners

  • Authors:

  • Alexandra Chouldechova;David Mease

  • Affiliations:

  • Stanford University, Stanford, CA, USA;Google, Mountain View, CA, USA

  • Venue:
  • Proceedings of the sixth ACM international conference on Web search and data mining
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

The query-document relevance judgments used in web search engine evaluation are traditionally provided by human assessors who have no particular association with the specific queries selected for the evaluation. Most commonly, queries are randomly sampled from search logs and in turn randomly assigned to the human assessors. In this paper, we consider a very different approach in which we instead ask the human assessors to provide their own queries from their recent search experiences. Using these queries as our sample, we compare the relevance judgments from the "owners" of the queries to the relevance judgments of the non-owners. We conduct experiments which reveal that query ownership has a substantial and beneficial impact on the accuracy of relevance judgments. In particular, we observe that owners are more consistently able to distinguish a higher quality set of search results from a lower quality set in a blind comparison. The implication for web search evaluation is that query owners provide more valuable relevance judgments than non-owners, presumably due to the background knowledge associated with their queries. We quantify the benefit of using owner assessments versus non-owner assessments in terms of sample size reduction. We also touch on some of the practical challenges associated with using query owners as assessors.