Introduction to the special issue on evaluating word sense disambiguation systems

  • Authors:
  • Philip Edmonds;Adam Kilgarriff

  • Affiliations:
  • Sharp Laboratories of Europe, Oxford Science Park, Oxford OX4 4GB, UK e-mail: phil@sharp.co.uk;Information Technology Research Institute, University of Brighton, Lewes Road, Brighton BN2 4GJ, UK e-mail: Adam.Kilgarriff@itri.brighton.ac.uk

  • Venue:
  • Natural Language Engineering
  • Year:
  • 2002

Quantified Score

Hi-index 0.00

Visualization

Abstract

Has system performance on Word Sense Disambiguation (WSD) reached a limit? Automatic systems don't perform nearly as well as humans on the task, and from the results of the SENSEVAL exercises, recent improvements in system performance appear negligible or even negative. Still, systems do perform much better than the baselines, so something is being done right. System evaluation is crucial to explain these results and to show the way forward. Indeed, the success of any project in WSD is tied to the evaluation methodology used, and especially to the formalization of the task that the systems perform. The evaluation of WSD has turned out to be as difficult as designing the systems in the first place.