Evaluating the effectiveness of relevance feedback based on a user simulation model: effects of a user scenario on cumulated gain value

  • Authors:

  • Heikki Keskustalo;Kalervo Järvelin;Ari Pirkola

  • Affiliations:

  • Department of Information Studies, University of Tampere, Tampere, Finland 33014;Department of Information Studies, University of Tampere, Tampere, Finland 33014;Department of Information Studies, University of Tampere, Tampere, Finland 33014

  • Venue:
  • Information Retrieval
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

We propose a method for performing evaluation of relevance feedback based on simulating real users. The user simulation applies a model defining the user's relevance threshold to accept individual documents as feedback in a graded relevance environment; user's patience to browse the initial list of retrieved documents; and his/her effort in providing the feedback. We evaluate the result by using cumulated gain-based evaluation together with freezing all documents seen by the user in order to simulate the point of view of a user who is browsing the documents during the retrieval process. We demonstrate the method by performing a simulation in the laboratory setting and present the "branching" curve sets characteristic for the presented evaluation method. Both the average and topic-by-topic results indicate that if the freezing approach is adopted, giving feedback of mixed quality makes sense for various usage scenarios even though the modeled users prefer finding especially the most relevant documents.