Evaluation of Open-Domain Question Answering Systems within a Time Constraint

  • Authors:

  • Elisa Noguera;Fernando Llopis;Antonio Ferrandez;Alberto Escapa

  • Affiliations:

  • University of Alicante, Spain;University of Alicante, Spain;University of Alicante, Spain;University of Alicante, Spain

  • Venue:
  • AINAW '07 Proceedings of the 21st International Conference on Advanced Information Networking and Applications Workshops - Volume 01
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

Previous works on evaluating the performance of Question Answering (QA) systems are focused in the evaluation of the precision. Nevertheless, the importance of the answer time never has been evaluated. This paper studies the problem of the evaluation in QA systems, focusing on the way in which the time can be considered. Specifically, we carried out an exercise for the evaluation of QA systems within a time constraint in the CLEF-2006 campaign, proposing new measures which combine the precision with the answer time. Also, the performance achieved by each participant system and statistical analysis of the results is given. The main conclusion is that the evaluation of QA systems in realtime can be a more realistic scenario than the tradicional used for the main evaluations forums in QA.