The TIPSTER SUMMAC Text Summarization Evaluation

Authors:
Inderjeet Mani;David House;Gary Klein;Lynette Hirschman;Therese Firmin;Beth Sundheim
Affiliations:
The MITRE Corporation, Reston, VA;The MITRE Corporation, Reston, VA;The MITRE Corporation, Reston, VA;The MITRE Corporation, Reston, VA;Department of Defense, Ft. Meade, MD;SPAWAR Systems Center, San Diego, CA
Venue:
EACL '99 Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics
Year:
1999

Citing 10
Cited 35

Constructing literature abstracts by computer: techniques and prospects

Information Processing and Management: an International Journal - Special issue on natural language processing and information retrieval
A trainable document summarizer

SIGIR '95 Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval
Automatic condensation of electronic publications by sentence selection

Information Processing and Management: an International Journal - Special issue: summarizing text
Generating summaries from event data

Information Processing and Management: an International Journal - Special issue: summarizing text
Automatic text structuring and summarization

Information Processing and Management: an International Journal - Special issue: methods and tools for the automatic construction of hypertext
Advantages of query biased summaries in information retrieval

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Variations in relevance judgments and the measurement of retrieval effectiveness

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
New Methods in Automatic Extracting

Journal of the ACM (JACM)
The reliability of a dialogue structure coding scheme

Computational Linguistics
Multi-document summarization by graph search and matching

AAAI'97/IAAI'97 Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence

Improving realism of topic tracking evaluation

SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
The Challenges of Automatic Summarization

Computer
Summarizing scientific articles: experiments with relevance and rhetorical status

Computational Linguistics - Summarization
Multi-document Summarization Using Informative Words and Its Evaluation with a QA System

CICLing '02 Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing
Examining the role of statistical and linguistic knowledge sources in a general-knowledge question-answering system

ANLC '00 Proceedings of the sixth conference on Applied natural language processing
The effects of analysing cohesion on document summarisation

COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
Improving summaries by revising them

ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
Glossary extraction and utilization in the information search and delivery system for IBM technical support

IBM Systems Journal
World wide web site summarization

Web Intelligence and Agent Systems
Web-page summarization using clickthrough data

Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Extracting key paragraph based on topic and event detection: towards multi-document summarization

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
Text summarizer in use: lessons learned from real world deployment and evaluation

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
A comparison of rankings produced by summarization evaluation measures

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
Multi-document summarization by visualizing topical content

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
Using summarization for automatic briefing generation

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
A practical text summarizer by paragraph extraction for Thai

AsianIR '03 Proceedings of the sixth international workshop on Information retrieval with Asian languages - Volume 11
Examining the consensus between human summaries: initial experiments with factoid analysis

HLT-NAACL-DUC '03 Proceedings of the HLT-NAACL 03 on Text summarization workshop - Volume 5
An information delivery system with automatic summarization for mobile commerce

Decision Support Systems
DUC in context

Information Processing and Management: an International Journal
Extrinsic Summarization Evaluation: A Decision Audit Task

MLMI '08 Proceedings of the 5th international workshop on Machine Learning for Multimodal Interaction
Extracting key paragraph based on topic and event detection: towards multi-document summarization

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization
Text summarizer in use: lessons learned from real world deployment and evaluation

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization
A comparison of rankings produced by summarization evaluation measures

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization
Multi-document summarization by visualizing topical content

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization
Using summarization for automatic briefing generation

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization
Extrinsic summarization evaluation: A decision audit task

ACM Transactions on Speech and Language Processing (TSLP)
Automatic summarization of MEDLINE citations for evidence-based medical treatment: A topic-oriented evaluation

Journal of Biomedical Informatics
Nouveau-rouge: A novelty metric for update summarization

Computational Linguistics
Pic-A-Topic: gathering information efficiently from recorded TV shows on travel

AIRS'06 Proceedings of the Third Asia conference on Information Retrieval Technology
Text summarisation in progress: a literature review

Artificial Intelligence Review
Impact of document structure on hierarchical summarization

ICADL'06 Proceedings of the 9th international conference on Asian Digital Libraries: achievements, Challenges and Opportunities
Multi-document summarization for e-learning

ICHL'09 Proceedings of the Second international conference on Hybrid Learning and Education
Evaluating web search result summaries

ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
Using the omega index for evaluating abstractive community detection

Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization
Juggling the Jigsaw: towards automated problem inference from network trouble tickets

nsdi'13 Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation

Quantified Score

Hi-index	0.00

Visualization

Abstract

The TIPSTER Text Summarization Evaluation (SUMMAC) has established definitively that automatic text summarization is very effective in relevance assessment tasks. Summaries as short as 17% of full text length sped up decision-making by almost a factor of 2 with no statistically significant degradation in F-score accuracy. SUMMAC has also introduced a new intrinsic method for automated evaluation of informative summaries.