An automatic method for summary evaluation using multiple evaluation results by a manual method

Authors:
Hidetsugu Nanba;Manabu Okumura
Affiliations:
Hiroshima City University, Hiroshima, Japan;Tokyo Institute of Technology, Yokohama, Japan
Venue:
COLING-ACL '06 Proceedings of the COLING/ACL on Main conference poster sessions
Year:
2006

Citing 5
Cited 4

A new quantitative quality measure for machine translation systems

COLING '92 Proceedings of the 14th conference on Computational linguistics - Volume 2
Applications of automatic evaluation methods to measuring a capability of speech translation system

EACL '03 Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics - Volume 1
Automatic evaluation of summaries using N-gram co-occurrence statistics

NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
A comparison of rankings produced by summarization evaluation measures

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
Kernel-based approach for automatic evaluation of natural language generation technologies: application to automatic summarization

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing

A new sentence similarity measure and sentence based extractive technique for automatic text summarization

Expert Systems with Applications: An International Journal
Intelligent location-based mobile news service system with automatic news summarization

Expert Systems with Applications: An International Journal
Fuzzy evolutionary optimization modeling and its applications to unsupervised categorization and extractive summarization

Expert Systems with Applications: An International Journal
Automatic evaluation of texts by using paraphrases

LTC'09 Proceedings of the 4th conference on Human language technology: challenges for computer science and linguistics

Quantified Score

Hi-index	0.00

Visualization

Abstract

To solve a problem of how to evaluate computer-produced summaries, a number of automatic and manual methods have been proposed. Manual methods evaluate summaries correctly, because humans evaluate them, but are costly. On the other hand, automatic methods, which use evaluation tools or programs, are low cost, although these methods cannot evaluate summaries as accurately as manual methods. In this paper, we investigate an automatic evaluation method that can reduce the errors of traditional automatic methods by using several evaluation results obtained manually. We conducted some experiments using the data of the Text Summarization Challenge 2 (TSC-2). A comparison with conventional automatic methods shows that our method outperforms other methods usually used.