In vitro evaluation of a program for machine-aided indexing

Authors:
Christian Jacquemin;Béatrice Daille;Jean Royanté;Xavier Polanco
Affiliations:
LIMSI-CNRS and University Paris 11, BP 133, 91403 Orsay Cedex, France;IRIN, Université de Nantes, 2, rue de la Houssiniére, BP 92208, 44322 Nantes Cedex 3, France;CNRS-INIST, Unité Recherche et Innovation, 54514 Vandoeuvre-lés-Nancy, France;-
Venue:
Information Processing and Management: an International Journal
Year:
2002

Citing 11
Cited 1

The state of retrieval system evaluation

Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
On the evaluation of IR systems

Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
Relevance reconsidered—towards an agenda for the 21st century: introduction to special topic issue on relevance research

Journal of the American Society for Information Science - Special issue: relevance research
Overview of the second text retrieval conference (TREC-2)

TREC-2 Proceedings of the second conference on Text retrieval conference
Natural language information retrieval

TREC-2 Proceedings of the second conference on Text retrieval conference
Relevance: the whole history

Journal of the American Society for Information Science - Special topic issue on the history of documentation and information science: part II
Computer Evaluation of Indexing and Text Processing

Journal of the ACM (JACM)
Information Retrieval

Information Retrieval
Introduction to Modern Information Retrieval

Introduction to Modern Information Retrieval
Accurate methods for the statistics of surprise and coincidence

Computational Linguistics - Special issue on using large corpora: I
Acquiring lexical generalizations from corpora: a case study for diathesis alternations

ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics

A symbolic approach to automatic multiword term structuring

Computer Speech and Language

Quantified Score

Hi-index	0.00

Visualization

Abstract

This article presents the human evaluation of ILIAD, a program for machine-aided indexing (MAI). It consists of two language engineering modules and is designed to assist expert librarians in computer-aided indexing and document analysis. Our aim is the expert evaluation of automatic multi-word term indexing. Evaluation is performed by documentary engineers. Cataloging and indexing are their principal tasks. They also have a good scientific knowledge of the domain to which the indexed documents belong.We first present the ILIAD program and the two systems submitted to this evaluation, the methodology (protocol) adopted, the differences between the protocol and the implementation, and the results of these evaluations. Human evaluation is divided into three parts: firstly the evaluation of controlled indexing, then free indexing and finally term variant extraction performed during controlled indexing. Finally, we analyze the relevance of this evaluation by calculating the agreement frequency and the Kappa coefficient and propose some future developments.