Interactive Evaluation of the Ostensive ModelUsing a New Test Collection of Images with Multiple Relevance Assessments

  • Authors:
  • Iain Campbell

  • Affiliations:
  • Computing Science, University of Glasgow, Glasgow, G12 8QQ, Scotland. iain@dcs.gla.ac.uk

  • Venue:
  • Information Retrieval
  • Year:
  • 2000

Quantified Score

Hi-index 0.00

Visualization

Abstract

The Ostensive Model proposes a manner of structuringthe uncertainty associated with individual relevancejudgements as sources of evidence in relevance feedback. Itproposes temporal profiles of uncertainty, motivating the applicationof a particular class of discount function with respect to the age ofthe evidence. This paper presents an initial evaluation of the relativeeffectiveness of different uncertainty discount functions.A novel direct manipulation interface to a multimediaretrieval system embodying the Ostensive Model is outlined briefly. Thepaper describes the construction andcharacteristics of a new image test collection utilising multiplebinary relevance assessments. The use of such multiple assessmentsand multiple interpretations of them are discussed. The evaluationenvironment is detailed in terms of theinterface, test collection, and tasks set to users. Multipleinterpretations of the results, and the statistical significanceof comparisons are presented.The results obtained in the evaluation are consistentwith the proposals of theOstensive Model—reinforcing a particular evidence profile. Theresults give clear pointers to further, more specific, evaluations.