Evaluation challenges in large-scale document summarization

Authors:
Dragomir R. Radev;Simone Teufel;Horacio Saggion;Wai Lam;John Blitzer;Hong Qi;Arda Çelebi;Danyu Liu;Elliott Drabek
Affiliations:
U. of Michigan;U. of Cambridge;U. of Sheffield;Chinese U. of Hong Kong;U. of Pennsylvania;U. of Michigan;USC/ISI;U. of Alabama;Johns Hopkins U.
Venue:
ACL '03 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1
Year:
2003

Citing 9
Cited 24

Automatic text processing

Automatic text processing
Text algorithms

Text algorithms
Automatic condensation of electronic publications by sentence selection

Information Processing and Management: an International Journal - Special issue: summarizing text
Assessing agreement on classification tasks: the kappa statistic

Computational Linguistics
Summarizing Similarities and Differences Among Related Documents

Information Retrieval
Generic summaries for indexing in information retrieval

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
A program for aligning sentences in bilingual corpora

Computational Linguistics - Special issue on using large corpora: I
SUMMAC: a text summarization evaluation

Natural Language Engineering
Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4

Centroid-based summarization of multiple documents

Information Processing and Management: an International Journal
A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Corpus and evaluation measures for multiple document summarization with multiple sources

COLING '04 Proceedings of the 20th international conference on Computational Linguistics
Semantic similarity applied to spoken dialogue summarization

COLING '04 Proceedings of the 20th international conference on Computational Linguistics
Using gene expression programming to construct sentence ranking functions for text summarization

COLING '04 Proceedings of the 20th international conference on Computational Linguistics
Generating gene summaries from biomedical literature: A study of semi-structured summarization

Information Processing and Management: an International Journal
Automatic summarising: The state of the art

Information Processing and Management: an International Journal
Learning query-biased web page summarization

Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Reviewing and Evaluating Automatic Term Recognition Techniques

GoTAL '08 Proceedings of the 6th international conference on Advances in Natural Language Processing
Abstraction summarization for managing the biomedical research literature

CLS '04 Proceedings of the HLT-NAACL Workshop on Computational Lexical Semantics
Automatic summarization of MEDLINE citations for evidence-based medical treatment: A topic-oriented evaluation

Journal of Biomedical Informatics
Automatically evaluating content selection in summarization without human models

EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
Summarizing short stories

Computational Linguistics
Focused multi-document summarization: human summarization activity vs. automated systems techniques

Journal of Computing Sciences in Colleges
Formal and functional assessment of the pyramid method for summary content evaluation*

Natural Language Engineering
Capturing user reading behaviors for personalized document summarization

Proceedings of the 16th international conference on Intelligent user interfaces
Multilingual summarization evaluation without human models

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
A semantic graph-based approach to biomedical summarisation

Artificial Intelligence in Medicine
Web Page Summarization for Just-in-Time Contextual Advertising

ACM Transactions on Intelligent Systems and Technology (TIST)
Text summarization and singular value decomposition

ADVIS'04 Proceedings of the Third international conference on Advances in Information Systems
Multiple documents summarization based on genetic algorithm

FSKD'06 Proceedings of the Third international conference on Fuzzy Systems and Knowledge Discovery
Extraction of relevant figures and tables for multi-document summarization

CICLing'12 Proceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part II
Degree centrality for semantic abstraction summarization of therapeutic studies

Journal of Biomedical Informatics
A genetic graph-based clustering approach to biomedical summarization

Proceedings of the 3rd International Conference on Web Intelligence, Mining and Semantics

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of (a) 100 Million automatic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese, (b) more than 10,000 manual abstracts and extracts, and (c) 200 Million automatic document and summary retrievals using 20 queries. We present both qualitative and quantitative results showing the strengths and draw-backs of all evaluation methods and how they rank the different summarizers.