Testing a decision-theoretic approach to the evaluation of information retrieval systems

  • Authors:

  • Ye Diana Wang;Guisseppi Forgionne

  • Affiliations:

  • Department of Applied Information Technology, GeorgeMason University, USA;Information Systems Department, University of MarylandBaltimore County, USA

  • Venue:
  • Journal of Information Science
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

With information overload a real problem, especially on the Internet, there has been much interest in developing effective and efficient information retrieval (IR) systems. The various information retrieval approaches will require accurate evaluation to justify the requisite substantial development and implementation investment. Recently, a comprehensive and integrated evaluation model has been proposed and illustrated. By analyzing the evaluation measures using the analytic hierarchy process (AHP), the model transforms IR evaluation into a multi-criteria decision making (MCDM) problem, which assesses both the IR outcome and the interactive IR process. This paper extends that research by refining the evaluation model and by testing the research question through mathematical testing and simulation. The tests confirm the need to include both process and outcome criteria in any IR evaluation and prove the superiority of the proposed decision-theoretic approach over the traditional evaluation methodologies that focus on the IR outcome alone.