Constructing literature abstracts by computer: techniques and prospects
Information Processing and Management: an International Journal - Special issue on natural language processing and information retrieval
A full-text retrieval system with a dynamic abstract generation function
SIGIR '94 Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval
A trainable document summarizer
SIGIR '95 Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval
Automatic condensation of electronic publications by sentence selection
Information Processing and Management: an International Journal - Special issue: summarizing text
Generating summaries from event data
Information Processing and Management: an International Journal - Special issue: summarizing text
Revision-based generation of natural language summaries providing historical background: corpus-based analysis, design, implementation and evaluation
Automatic text structuring and summarization
Information Processing and Management: an International Journal - Special issue: methods and tools for the automatic construction of hypertext
Advantages of query biased summaries in information retrieval
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Variations in relevance judgments and the measurement of retrieval effectiveness
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Machine learning of generic and user-focused summarization
AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Summarizing text documents: sentence selection and evaluation metrics
Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
The decomposition of human-written summary sentences
Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
The automatic construction of large-scale corpora for summarization research
Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
Training a selection function for extraction
Proceedings of the eighth international conference on Information and knowledge management
New Methods in Automatic Extracting
Journal of the ACM (JACM)
Summarizing Similarities and Differences Among Related Documents
Information Retrieval
Evaluating Natural Language Processing Systems: An Analysis and Review
Evaluating Natural Language Processing Systems: An Analysis and Review
Advances in Automatic Text Summarization
Advances in Automatic Text Summarization
Introduction to Modern Information Retrieval
Introduction to Modern Information Retrieval
Statistics-Based Summarization - Step One: Sentence Compression
Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence
Lexical cohesion computed by thesaural relations as an indicator of the structure of text
Computational Linguistics
The reliability of a dialogue structure coding scheme
Computational Linguistics
TextTiling: segmenting text into multi-paragraph subtopic passages
Computational Linguistics
Generating natural language summaries from multiple on-line sources
Computational Linguistics - Special issue on natural language generation
Message Understanding Conference-6: a brief history
COLING '96 Proceedings of the 16th conference on Computational linguistics - Volume 1
Improving summaries by revising them
ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
Concept identification and presentation in the context of technical text summarization
NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
A comparison of rankings produced by summarization evaluation measures
NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
Multi-document summarization by graph search and matching
AAAI'97/IAAI'97 Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence
Natural language question answering: the view from here
Natural Language Engineering
Evaluation challenges in large-scale document summarization
ACL '03 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1
Choosing words in computer-generated weather forecasts
Artificial Intelligence - Special volume on connecting language to the world
Corpus and evaluation measures for multiple document summarization with multiple sources
COLING '04 Proceedings of the 20th international conference on Computational Linguistics
Choosing the content of textual summaries of large time-series data sets
Natural Language Engineering
Automatic summarising: The state of the art
Information Processing and Management: an International Journal
Task-based evaluation of text summarization using Relevance Prediction
Information Processing and Management: an International Journal
Supervised automatic evaluation for summarization with voted regression model
Information Processing and Management: an International Journal
User-model based personalized summarization
Information Processing and Management: an International Journal
Information Processing and Management: an International Journal
What are meeting summaries?: an analysis of human extractive summaries in meeting corpus
SIGdial '08 Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue
Evalinitiatives '03 Proceedings of the EACL 2003 Workshop on Evaluation Initiatives in Natural Language Processing: are evaluation methods, metrics and resources reusable?
Choosing words in computer-generated weather forecasts
Artificial Intelligence - Special volume on connecting language to the world
Saliency Regions for 3D Mesh Abstraction
PCM '09 Proceedings of the 10th Pacific Rim Conference on Multimedia: Advances in Multimedia Information Processing
Formal and functional assessment of the pyramid method for summary content evaluation*
Natural Language Engineering
Automated skimming in response to questions for nonvisual readers
SLPAT '10 Proceedings of the NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies
An approach to generate indicative summaries for Japanese documents
Proceedings of the 1st International Conference on Intelligent Semantic Web-Services and Applications
Multilingual summarization evaluation without human models
COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Discourse constraints for document compression
Computational Linguistics
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts of ACL 2011
Evaluating sentence compression: pitfalls and suggested remedies
MTTG '11 Proceedings of the Workshop on Monolingual Text-To-Text Generation
Text summarisation in progress: a literature review
Artificial Intelligence Review
Computer Speech and Language
Automatic extractive text summarization based on fuzzy logic: a sentence oriented approach
SEMCCO'11 Proceedings of the Second international conference on Swarm, Evolutionary, and Memetic Computing - Volume Part I
Hi-index | 0.00 |
The TIPSTER Text Summarization Evaluation (SUMMAC) has developed several new extrinsic and intrinsic methods for evaluating summaries. It has established definitively that automatic text summarization is very effective in relevance assessment tasks on news articles. Summaries as short as 17% of full text length sped up decision-making by almost a factor of 2 with no statistically significant degradation in accuracy. Analysis of feedback forms filled in after each decision indicated that the intelligibility of present-day machine-generated summaries is high. Systems that performed most accurately in the production of indicative and informative topic-related summaries used term frequency and co-occurrence statistics, and vocabulary overlap comparisons between text passages. However, in the absence of a topic, these statistical methods do not appear to provide any additional leverage: in the case of generic summaries, the systems were indistinguishable in accuracy. The paper discusses some of the tradeoffs and challenges faced by the evaluation, and also lists some of the lessons learned, impacts, and possible future directions. The evaluation methods used in the SUMMAC evaluation are of interest to both summarization evaluation as well as evaluation of other ‘output-related’ NLP technologies, where there may be many potentially acceptable outputs, with no automatic way to compare them.