Automatic evaluation of summaries using N-gram co-occurrence statistics

Authors:
Chin-Yew Lin;Eduard Hovy
Affiliations:
University of Southern California, Marina del Rey, CA;University of Southern California, Marina del Rey, CA
Venue:
NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Year:
2003

Citing 5
Cited 315

Evaluating Natural Language Processing Systems: An Analysis and Review

Evaluating Natural Language Processing Systems: An Analysis and Review
NewsInEssence: a system for domain-independent, real-time news clustering and multi-document summarization

HLT '01 Proceedings of the first international conference on Human language technology research
A comparison of rankings produced by summarization evaluation measures

NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4
Manual and automatic evaluation of summaries

AS '02 Proceedings of the ACL-02 Workshop on Automatic Summarization - Volume 4
Tracking and summarizing news on a daily basis with Columbia's Newsblaster

HLT '02 Proceedings of the second international conference on Human Language Technology Research

Cross-lingual C*ST*RD: English access to Hindi information

ACM Transactions on Asian Language Information Processing (TALIP)
Cross-language headline generation for Hindi

ACM Transactions on Asian Language Information Processing (TALIP)
Evaluation of an extraction-based approach to answering definitional questions

Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
A hybrid statistical/linguistic model for generating news story gists

Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
Developing practical automatic metadata assignment and evaluation tools for internet resources

Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries
Book Review, Inderjeet Mani: Automatic Summarization, John Benjamins Publishing Co., Amsterdam, The Netherlands, 2001, xi + 286 pp.

Machine Translation
Web-page summarization using clickthrough data

Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Do summaries help?

Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Generic soft pattern models for definitional question answering

Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Sentence Fusion for Multidocument News Summarization

Computational Linguistics
Improving summarization performance by sentence compression: a pilot study

AsianIR '03 Proceedings of the sixth international workshop on Information retrieval with Asian languages - Volume 11
The potential and limitations of automatic sentence extraction for summarization

HLT-NAACL-DUC '03 Proceedings of the HLT-NAACL 03 on Text summarization workshop - Volume 5
Automatic question answering using the web: Beyond the Factoid

Information Retrieval
A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Concept frequency distribution in biomedical text summarization

CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
From social bookmarking to social summarization: an experiment in community-based summary generation

Proceedings of the 12th international conference on Intelligent user interfaces
An empirical study of information synthesis tasks

ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
A unified framework for automatic evaluation using N-gram co-occurrence statistics

ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
Graph-based ranking algorithms for sentence extraction, applied to text summarization

ACLdemo '04 Proceedings of the ACL 2004 on Interactive poster and demonstration sessions
Modeling local coherence: an entity-based approach

ACL '05 Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics
QARLA: a framework for the evaluation of text summarization systems

ACL '05 Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics
Extractive summarization using inter- and intra- event relevance

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
A formal model for information selection in multi-sentence text extraction

COLING '04 Proceedings of the 20th international conference on Computational Linguistics
Syntactic simplification for improving content selection in multi-document summarization

COLING '04 Proceedings of the 20th international conference on Computational Linguistics
Improving multilingual summarization: using redundancy in the input to correct MT errors

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Kernel-based approach for automatic evaluation of natural language generation technologies: application to automatic summarization

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Bayesian learning in text summarization

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Combining deep linguistics analysis and surface pattern learning: a hybrid approach to Chinese definitional question answering

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Automatically evaluating answers to definition questions

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Incorporating speaker and discourse features into speech summarization

HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
Will pyramids built of nuggets topple over?

HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
ParaEval: using paraphrases to evaluate summaries automatically

HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
An information-theoretic approach to automatic evaluation of summaries

HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
Language independent extractive summarization

ACLdemo '05 Proceedings of the ACL 2005 on Interactive poster and demonstration sessions
Soft pattern matching models for definitional question answering

ACM Transactions on Information Systems (TOIS)
Automatic Evaluation of Information Ordering: Kendall's Tau

Computational Linguistics
Using Cross-Document Random Walks for Topic-Focused Multi-Document

WI '06 Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence
An automatic method for summary evaluation using multiple evaluation results by a manual method

COLING-ACL '06 Proceedings of the COLING/ACL on Main conference poster sessions
CollabSum: exploiting multiple document clustering for collaborative single document summarizations

SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
The use of domain-specific concepts in biomedical text summarization

Information Processing and Management: an International Journal
Generating gene summaries from biomedical literature: A study of semi-structured summarization

Information Processing and Management: an International Journal
Task-based evaluation of text summarization using Relevance Prediction

Information Processing and Management: an International Journal
Older versions of the ROUGEeval summarization evaluation system were easier to fool

Information Processing and Management: an International Journal
Satisfying information needs with multi-document summaries

Information Processing and Management: an International Journal
Supervised automatic evaluation for summarization with voted regression model

Information Processing and Management: an International Journal
Multi-candidate reduction: Sentence compression as a tool for document summarization tasks

Information Processing and Management: an International Journal
Two uses of anaphora resolution in summarization

Information Processing and Management: an International Journal
User-model based personalized summarization

Information Processing and Management: an International Journal
Syntactic sentence compression in the biomedical domain: facilitating access to related articles

Information Retrieval
Open-domain question: answering

Foundations and Trends in Information Retrieval
Modeling local coherence: An entity-based approach

Computational Linguistics
Categorizing and ranking search engine's results by semantic similarity

Proceedings of the 2nd international conference on Ubiquitous information management and communication
Biomedical text summarisation using concept chains

International Journal of Data Mining and Bioinformatics
Semantic text similarity using corpus-based word similarity and string similarity

ACM Transactions on Knowledge Discovery from Data (TKDD)
iSpreadRank: Ranking sentences for extraction-based summarization using feature weight propagation in the sentence similarity network

Expert Systems with Applications: An International Journal
Applications of corpus-based semantic similarity and word segmentation to database schema matching

The VLDB Journal — The International Journal on Very Large Data Bases
Query-sensitive mutual reinforcement chain and its application in query-oriented multi-document summarization

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Comments-oriented document summarization: understanding documents with readers' feedback

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Multi-document summarization using cluster-based link analysis

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Multi-document summarization via sentence-level semantic analysis and symmetric matrix factorization

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
TSCAN: a novel method for topic summarization and content anatomy

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Latent dirichlet allocation based multi-document summarization

Proceedings of the second workshop on Analytics for noisy unstructured text data
Summarization system evaluation revisited: N-gram graphs

ACM Transactions on Speech and Language Processing (TSLP)
Question Answering Summarization of Multiple Biomedical Documents

CAI '07 Proceedings of the 20th conference of the Canadian Society for Computational Studies of Intelligence on Advances in Artificial Intelligence
An n-Gram and Initial Description Based Approach for Entity Ranking Track

Focused Access to XML Documents
Combining Multiple Features for Automatic Text Summarization through Machine Learning

PROPOR '08 Proceedings of the 8th international conference on Computational Processing of the Portuguese Language
Meta Comments for Summarizing Meeting Speech

MLMI '08 Proceedings of the 5th international workshop on Machine Learning for Multimodal Interaction
Web content summarization using social bookmarks: a new approach for social summarization

Proceedings of the 10th ACM workshop on Web information and data management
Using query expansion in graph-based approach for query-focused multi-document summarization

Information Processing and Management: an International Journal
Biased LexRank: Passage retrieval using random walks with question-based priors

Information Processing and Management: an International Journal
Text Summarization by Sentence Extraction Using Unsupervised Learning

MICAI '08 Proceedings of the 7th Mexican International Conference on Artificial Intelligence: Advances in Artificial Intelligence
An Ontology-Based Approach to Text Summarization

WI-IAT '08 Proceedings of the 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology - Volume 03
Document-Based HITS Model for Multi-document Summarization

PRICAI '08 Proceedings of the 10th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence
Exploiting the Role of Named Entities in Query-Oriented Document Summarization

PRICAI '08 Proceedings of the 10th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence
A new sentence similarity measure and sentence based extractive technique for automatic text summarization

Expert Systems with Applications: An International Journal
Enhancing diversity, coverage and balance for summarization through structure learning

Proceedings of the 18th international conference on World wide web
A light-weight summarizer based on language model with relative entropy

Proceedings of the 2009 ACM symposium on Applied Computing
Estimating Risk of Picking a Sentence for Document Summarization

CICLing '09 Proceedings of the 10th International Conference on Computational Linguistics and Intelligent Text Processing
Query-Focused Summarization by Combining Topic Model and Affinity Propagation

APWeb/WAIM '09 Proceedings of the Joint International Conferences on Advances in Data and Web Management
Query-Oriented Summarization Based on Neighborhood Graph Model

ICCPOL '09 Proceedings of the 22nd International Conference on Computer Processing of Oriental Languages. Language Technology for the Knowledge-based Economy
Learning Similarity Functions in Graph-Based Document Summarization

ICCPOL '09 Proceedings of the 22nd International Conference on Computer Processing of Oriental Languages. Language Technology for the Knowledge-based Economy
Event-Based Summarization Using Critical Temporal Event Term Chain

ICCPOL '09 Proceedings of the 22nd International Conference on Computer Processing of Oriental Languages. Language Technology for the Knowledge-based Economy
Cover Coefficient-Based Multi-document Summarization

ECIR '09 Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval
Search-based structured prediction

Machine Learning
Ontology-based relevance analysis for automatic reference tracking

International Journal of Computer Applications in Technology
Query-based sentence fusion is better defined and leads to more preferred results than generic sentence fusion

HLT-Short '08 Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers
Intrinsic vs. extrinsic evaluation measures for referring expression generation

HLT-Short '08 Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers
Extractive summarization based on event term clustering

ACL '07 Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions
Investigations on event-based summarization

COLING ACL '06 Proceedings of the 21st International Conference on computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
Automatic summary assessment for intelligent tutoring systems

Computers & Education
Towards automatic generation of gene summary

BioNLP '09 Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing
An approach to text summarization

CLIAWS3 '09 Proceedings of the Third International Workshop on Cross Lingual Information Access: Addressing the Information Need of Multilingual Societies
Extrinsic summarization evaluation: A decision audit task

ACM Transactions on Speech and Language Processing (TSLP)
Corpus-based and knowledge-based measures of text semantic similarity

AAAI'06 Proceedings of the 21st national conference on Artificial intelligence - Volume 1
Pedagogically useful extractive summaries for science education

COLING '08 Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1
PNR2: ranking sentences with positive and negative reinforcement for query-oriented update summarization

COLING '08 Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1
Extractive summarization using supervised and semi-supervised learning

COLING '08 Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1
Company-oriented extractive summarization of financial news

EACL '09 Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics
Sentiment summarization: evaluating and learning user preferences

EACL '09 Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics
Text summarization model based on maximum coverage problem and its variant

EACL '09 Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics
Re-evaluating machine translation results with paraphrase support

EMNLP '06 Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing
Seed and Grow: augmenting statistically generated summary sentences using schematic word patterns

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
An exploration of document impact on graph-based multi-document summarization

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Using N-Grams to understand the nature of summaries

HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers
Improved affinity graph based multi-document summarization

NAACL-Short '06 Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers
A semi-automatic evaluation scheme: automated nuggetization for manual annotation

NAACL-Short '07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers
Text comparison using machine-generated nuggets

NAACL-Demonstrations '07 Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations
Language independent extractive summarization

AAAI'05 Proceedings of the 20th national conference on Artificial intelligence - Volume 4
Single document summarization with document expansion

AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 1
A text-to-picture synthesis system for augmenting communication

AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 2
What are meeting summaries?: an analysis of human extractive summaries in meeting corpus

SIGdial '08 Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue
LexRank: graph-based lexical centrality as salience in text summarization

Journal of Artificial Intelligence Research
A predictive approach to help-desk response generation

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Multi-document summarization by maximizing informative content-words

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Document summarization using conditional random fields

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Manifold-ranking based topic-focused multi-document summarization

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Measuring the semantic similarity of texts

EMSEE '05 Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment
An Unsupervised Model of Exploiting the Web to Answer Definitional Questions

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 01
Summarizing Documents by Measuring the Importance of a Subset of Vertices within a Graph

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 01
Automatic evaluation of text coherence: models and representations

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
iRANK: an interactive ranking framework and its application in query-focused summarization

Proceedings of the 18th ACM conference on Information and knowledge management
Dimensionality reduction aids term co-occurrence based multi-document summarization

SumQA '06 Proceedings of the Workshop on Task-Focused Summarization and Question Answering
Automating help-desk responses: a comparative study of information-gathering approaches

SumQA '06 Proceedings of the Workshop on Task-Focused Summarization and Question Answering
Graph-based multi-modality learning for topic-focused multi-document summarization

IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
Leveraging structural relations for fluent compressions at multiple compression rates

ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
An integrated multi-document summarization approach based on word hierarchical representation

ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
Co-feedback ranking for query-focused summarization

ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
Multi-document summarization using sentence-based topic models

ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
An investigation into the validity of some metrics for automatically evaluating natural language generation systems

Computational Linguistics
An empirical study of corpus-based response automation methods for an e-mail-based help-desk domain

Computational Linguistics
Extractive Summarization Based on Event Term Temporal Relation Graph and Critical Chain

AIRS '09 Proceedings of the 5th Asia Information Retrieval Symposium on Information Retrieval Technology
Automatically evaluating content selection in summarization without human models

EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
A statistical service composition approach based on hidden hierarchy Markov model

ICACT'09 Proceedings of the 11th international conference on Advanced Communication Technology - Volume 1
Lessons learned from large scale evaluation of systems that produce text: nightmares and pleasant surprises

INLG '06 Proceedings of the Fourth International Natural Language Generation Conference
Reducing redundancy in multi-document summarization using lexical semantic similarity

UCNLG+Sum '09 Proceedings of the 2009 Workshop on Language Generation and Summarisation
Extractive vs. NLG-based abstractive summarization of evaluative text: the effect of corpus controversiality

INLG '08 Proceedings of the Fifth International Natural Language Generation Conference
Automated metrics that agree with human judgements on generated output for an embodied conversational agent

INLG '08 Proceedings of the Fifth International Natural Language Generation Conference
Automatic generation of bid phrases for online advertising

Proceedings of the third ACM international conference on Web search and data mining
Summarizing short stories

Computational Linguistics
A document-sensitive graph model for multi-document summarization

Knowledge and Information Systems
Exploiting neighborhood knowledge for single document summarization and keyphrase extraction

ACM Transactions on Information Systems (TOIS)
Studies on intrinsic summary evaluation

International Journal of Artificial Intelligence and Soft Computing
Genetic algorithm based multi-document summarization

PRICAI'06 Proceedings of the 9th Pacific Rim international conference on Artificial intelligence
A study of global inference algorithms in multi-document summarization

ECIR'07 Proceedings of the 29th European conference on IR research
A comprehensive comparative evaluation of RST-based summarization methods

ACM Transactions on Speech and Language Processing (TSLP)
CONANN: an online biomedical concept annotator

DILS'07 Proceedings of the 4th international conference on Data integration in the life sciences
Arabic/English multi-document summarization with CLASSY: the past and the future

CICLing'08 Proceedings of the 9th international conference on Computational linguistics and intelligent text processing
Lexical cohesion based topic modeling for summarization

CICLing'08 Proceedings of the 9th international conference on Computational linguistics and intelligent text processing
Terms derived from frequent sequences for extractive text summarization

CICLing'08 Proceedings of the 9th international conference on Computational linguistics and intelligent text processing
A cluster-sensitive graph model for query-oriented multi-document summarization

ECIR'08 Proceedings of the IR research, 30th European conference on Advances in information retrieval
Web news summarization via soft clustering algorithm

FSKD'09 Proceedings of the 6th international conference on Fuzzy systems and knowledge discovery - Volume 7
Emotion and behaviour in automatic dialogue summarisation

Companion Proceedings of the XIV Brazilian Symposium on Multimedia and the Web
Automatic summarization for text simplification: evaluating text understanding by poor readers

Companion Proceedings of the XIV Brazilian Symposium on Multimedia and the Web
Intertopic information mining for query-based summarization

Journal of the American Society for Information Science and Technology
Paraphrase identification using machine learning techniques

ICNVS'10 Proceedings of the 12th international conference on Networking, VLSI and signal processing
Semantic Network Language Generation based on a Semantic Networks Serialization Grammar

World Wide Web
EUSUM: extracting easy-to-understand english summaries for non-native readers

Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Blog snippets: a comments-biased approach

Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Feature subset non-negative matrix factorization and its applications to document understanding

Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Many are better than one: improving multi-document summarization via weighted consensus

Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
VisualSum: an interactive multi-document summarizationsystem using visualization

Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
DivRank: the interplay of prestige and diversity in information networks

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Constructing query-biased summaries: a comparison of human and system generated snippets

Proceedings of the third symposium on Information interaction in context
An improved web information summarization based on SSSC

CAR'10 Proceedings of the 2nd international Asia conference on Informatics in control, automation and robotics - Volume 3
Summarizing microblogs automatically

HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Automatic evaluation of linguistic quality in multi-document summarization

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Automatic generation of story highlights

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
A hybrid hierarchical model for multi-document summarization

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Cross-language document summarization based on machine translation quality prediction

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
A new approach to improving multilingual summarization using a genetic algorithm

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Wrapping up a summary: from representation to generation

ACLShort '10 Proceedings of the ACL 2010 Conference Short Papers
From bursty patterns to bursty facts: The effectiveness of temporal text mining for news

Proceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence
Social summarization in collaborative web search

Information Processing and Management: an International Journal
Automated skimming in response to questions for nonvisual readers

SLPAT '10 Proceedings of the NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies
CONE: metrics for automatic evaluation of named entity co-reference resolution

NEWS '10 Proceedings of the 2010 Named Entities Workshop
Eliminating redundancy by spectral relaxation for multi-document summarization

TextGraphs-5 Proceedings of the 2010 Workshop on Graph-based Methods for Natural Language Processing
Title generation with quasi-synchronous grammar

EMNLP '10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing
Automatic evaluation of translation quality for distant language pairs

EMNLP '10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing
Document update summarization using incremental hierarchical clustering

CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Topic aspect analysis for multi-document summarization

CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Textual properties and task based evaluation: investigating the role of surface properties, structure and content

INLG '10 Proceedings of the 6th International Natural Language Generation Conference
Opinosis: a graph-based approach to abstractive summarization of highly redundant opinions

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics
Evaluating N-gram based evaluation metrics for automatic keyphrase extraction

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics
Text summarization of Turkish texts using latent semantic analysis

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics
Multi-document summarization via the minimum dominating set

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics
SemanticRank: ranking keywords and sentences using semantic graphs

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics
Towards a unified approach to simultaneous single-document and multi-document summarizations

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics
VERT: automatic evaluation of video summaries

Proceedings of the international conference on Multimedia
The automatic assessment of free text answers using a modified BLEU algorithm

Computers & Education
Structural features for predicting the linguistic quality of text: applications to machine translation, automatic summarization and human-authored text

Empirical methods in natural language generation
Introducing shared tasks to NLG: the TUNA shared task evaluation challenges

Empirical methods in natural language generation
CorrRank: update summarization based on topic correlation analysis

ICIC'10 Proceedings of the Advanced intelligent computing theories and applications, and 6th international conference on Intelligent computing
Using parallel corpora for multilingual (multi-document) summarisation evaluation

CLEF'10 Proceedings of the 2010 international conference on Multilingual and multimodal information access evaluation: cross-language evaluation forum
Identification of rhetorical roles for segmentation and summarization of a legal judgment

Artificial Intelligence and Law
Comparative evaluation of ontology-based Automatic Reference Tracking (ART)

International Journal of Networking and Virtual Organisations
Automatic categorization and summarization of documentaries

Journal of Information Science
A mining technique using n-grams and motion transcripts for body sensor network data repository

WH '10 Wireless Health 2010
Enhancement of subjective logic for semantic document analysis using hierarchical document signature

ICONIP'10 Proceedings of the 17th international conference on Neural information processing: theory and algorithms - Volume Part I
Discourse indicators for content selection in summarization

SIGDIAL '10 Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Learning summary content units with topic modeling

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Learning to model domain-specific utterance sequences for extractive summarization of contact center dialogues

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
A comparative study on ranking and selection strategies for multi-document summarization

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Multilingual summarization evaluation without human models

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
A method for automatically generating a mediatory summary to verify credibility of information on the web

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
A new approach for multi-document update summarization

Journal of Computer Science and Technology
Fuzzy evolutionary optimization modeling and its applications to unsupervised categorization and extractive summarization

Expert Systems with Applications: An International Journal
A novel approach for research paper abstracts summarization using cluster based sentence extraction

Proceedings of the International Conference & Workshop on Emerging Trends in Technology
Aspect-based extractive summarization of online reviews

Proceedings of the 2011 ACM Symposium on Applied Computing
Automatic evaluation of texts by using paraphrases

LTC'09 Proceedings of the 4th conference on Human language technology: challenges for computer science and linguistics
Summarization of personal photologs using multidimensional content and context

Proceedings of the 1st ACM International Conference on Multimedia Retrieval
Integrating Document Clustering and Multidocument Summarization

ACM Transactions on Knowledge Discovery from Data (TKDD)
Using bilingual information for cross-language document summarization

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Comparative news summarization using linear programming

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2
Social context summarization

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Summarizing the differences in multilingual news

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Evolutionary timeline summarization: a balanced optimization framework via iterative substitution

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Summarizing decisions in spoken meetings

WASDGML '11 Proceedings of the Workshop on Automatic Summarization for Different Genres, Media, and Languages
Text summarization using Latent Semantic Analysis

Journal of Information Science
Improving graph-based random walks for complex question answering using syntactic, shallow semantic and extended string subsequence kernels

Information Processing and Management: an International Journal
iDVS: an interactive multi-document visual summarization system

ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part III
Towards effective event detection, tracking and summarization on microblog data

WAIM'11 Proceedings of the 12th international conference on Web-age information management
A novel approach to update summarization using evolutionary manifold-ranking and spectral clustering

Expert Systems with Applications: An International Journal
A hybrid sentence ordering strategy in multi-document summarization

WISE'06 Proceedings of the 7th international conference on Web Information Systems
The great importance of cross-document relationships for multi-document summarization

ICCPOL'06 Proceedings of the 21st international conference on Computer Processing of Oriental Languages: beyond the orient: the research challenges ahead
Building document graphs for multiple news articles summarization: an event-based approach

ICCPOL'06 Proceedings of the 21st international conference on Computer Processing of Oriental Languages: beyond the orient: the research challenges ahead
Multilingual machine translation of closed captions for digital television with dynamic dictionary adaptation

ICCPOL'06 Proceedings of the 21st international conference on Computer Processing of Oriental Languages: beyond the orient: the research challenges ahead
Paraphrase identification on the basis of supervised machine learning techniques

FinTAL'06 Proceedings of the 5th international conference on Advances in Natural Language Processing
Using bleu-like algorithms for the automatic recognition of entailment

MLCW'05 Proceedings of the First international conference on Machine Learning Challenges: evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment
Experiments on statistical and pattern-based biographical summarization

EPIA'05 Proceedings of the 12th Portuguese conference on Progress in Artificial Intelligence
Text specificity and impact on quality of news summaries

MTTG '11 Proceedings of the Workshop on Monolingual Text-To-Text Generation
Incorporating cross-document relationships between sentences for single document summarizations

ECDL'06 Proceedings of the 10th European conference on Research and Advanced Technology for Digital Libraries
A hybrid model for sentence ordering in extractive multi-document summarization

AIRS'06 Proceedings of the Third Asia conference on Information Retrieval Technology
Multi-document summarization using a clustering-based hybrid strategy

AIRS'06 Proceedings of the Third Asia conference on Information Retrieval Technology
Random walks on text structures

CICLing'06 Proceedings of the 7th international conference on Computational Linguistics and Intelligent Text Processing
Multi-document summarization based on BE-Vector clustering

CICLing'06 Proceedings of the 7th international conference on Computational Linguistics and Intelligent Text Processing
Deriving event relevance from the ontology constructed with formal concept analysis

CICLing'06 Proceedings of the 7th international conference on Computational Linguistics and Intelligent Text Processing
Computational generation of referring expressions: A survey

Computational Linguistics
Enhancing n-gram-based summary evaluation using information content and a taxonomy

ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
LexTrim: a lexical cohesion based approach to parse-and-trim style headline generation

CICLing'05 Proceedings of the 6th international conference on Computational Linguistics and Intelligent Text Processing
Capture of evidence for summarization: an application of enhanced subjective logic

PAKDD'10 Proceedings of the 14th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part II
Timeline generation through evolutionary trans-temporal summarization

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Corpus-guided sentence generation of natural images

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Ranking human and machine summarization systems

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Generating aspect-oriented multi-document summarization with event-aspect model

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Summarize what you are interested in: an optimization framework for interactive personalized summarization

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Using word sequences for text summarization

TSD'06 Proceedings of the 9th international conference on Text, Speech and Dialogue
Filtering for profile-biased multi-document summarization

ECIR'05 Proceedings of the 27th European conference on Advances in Information Retrieval Research
Comparing topiary-style approaches to headline generation

ECIR'05 Proceedings of the 27th European conference on Advances in Information Retrieval Research
Modeling and evaluating summaries using complex networks

PROPOR'06 Proceedings of the 7th international conference on Computational Processing of the Portuguese Language
A comparative study of information-gathering approaches for answering help-desk email inquiries

AI'06 Proceedings of the 19th Australian joint conference on Artificial Intelligence: advances in Artificial Intelligence
GEMS: generative modeling for evaluation of summaries

CICLing'10 Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing
Sentence ordering in extractive MDS

IDEAL'06 Proceedings of the 7th international conference on Intelligent Data Engineering and Automated Learning
EM clustering algorithm for automatic text summarization

MICAI'11 Proceedings of the 10th Mexican international conference on Advances in Artificial Intelligence - Volume Part I
A text summarizer for Arabic

Computer Speech and Language
Weighted consensus multi-document summarization

Information Processing and Management: an International Journal
Toward a Unified Framework for Standard and Update Multi-Document Summarization

ACM Transactions on Asian Language Information Processing (TALIP)
Improving document summarization by incorporating social contextual information

AIRS'11 Proceedings of the 7th Asia conference on Information Retrieval Technology
Using graph based mapping of co-occurring words and closeness centrality score for summarization evaluation

CICLing'12 Proceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part II
Combining syntax and semantics for automatic extractive single-document summarization

CICLing'12 Proceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part II
Using wikipedia anchor text and weighted clustering coefficient to enhance the traditional multi-document summarization

CICLing'12 Proceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part II
Multi-document summarization exploiting frequent itemsets

Proceedings of the 27th Annual ACM Symposium on Applied Computing
Graph-based methods for multi-document summarization: exploring relationship maps, complex networks and discourse information

PROPOR'12 Proceedings of the 10th international conference on Computational Processing of the Portuguese Language
A behavioural mode research on user-focus summarization

Mathematical and Computer Modelling: An International Journal
Learning the "Whys": Discovering design rationale using text mining - An algorithm perspective

Computer-Aided Design
Fast bregman divergence NMF using taylor expansion and coordinate descent

Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Hierarchical graph summarization: leveraging hybrid information through visible and invisible linkage

PAKDD'12 Proceedings of the 16th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part II
GenDocSum+MCLR: Generic document summarization based on maximum coverage and less redundancy

Expert Systems with Applications: An International Journal
A unified graph model for Chinese product review summarization using richer information

Proceedings of the First International Workshop on Issues of Sentiment Discovery and Opinion Mining
Multi-aspect query summarization by composite query

SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Comparative document summarization via discriminative sentence selection

ACM Transactions on Knowledge Discovery from Data (TKDD)
SumView: A Web-based engine for summarizing product reviews and customer opinions

Expert Systems with Applications: An International Journal
Large-margin learning of submodular summarization models

EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics
Challenges and solutions in the opinion summarization of user-generated content

Journal of Intelligent Information Systems
TakeLab: systems for measuring semantic text similarity

SemEval '12 Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation
UNT: a supervised synergistic approach to semantic text similarity

SemEval '12 Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation
CDDS: Constraint-driven document summarization models

Expert Systems with Applications: An International Journal
Combining coherence models and machine translation evaluation metrics for summarization evaluation

ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1
Multiple aspect summarization using integer linear programming

EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning
Answering opinion questions on products by exploiting hierarchical organization of consumer reviews

EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning
An assessment of the accuracy of automatic evaluation in summarization

Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization
Towards a unified approach based on affinity graph to various multi-document summarizations

ECDL'07 Proceedings of the 11th European conference on Research and Advanced Technology for Digital Libraries
Unsupervised topic modeling approaches to decision summarization in spoken meetings

SIGDIAL '12 Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Focused meeting summarization via unsupervised relation extraction

SIGDIAL '12 Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Non-syntactic word prediction for AAC

SLPAT '12 Proceedings of the Third Workshop on Speech and Language Processing for Assistive Technologies
Visualizing timelines: evolutionary summarization via iterative reinforcement between text and image streams

Proceedings of the 21st ACM international conference on Information and knowledge management
DESAMC+DocSum: Differential evolution with self-adaptive mutation and crossover parameters for multi-document summarization

Knowledge-Based Systems
Joint topic modeling for event summarization across news and social media streams

Proceedings of the 21st ACM international conference on Information and knowledge management
A progressive sentence selection strategy for document summarization

Information Processing and Management: an International Journal
Generic multi-document summarization using topic-oriented information

PRICAI'12 Proceedings of the 12th Pacific Rim international conference on Trends in Artificial Intelligence
Localised topic information extraction for summarisation using syntactic sequences

International Journal of Knowledge and Web Intelligence
Formulation of document summarization as a 0-1 nonlinear programming problem

Computers and Industrial Engineering
From image annotation to image description

ICONIP'12 Proceedings of the 19th international conference on Neural Information Processing - Volume Part V
Translating related words to videos and back through latent topics

Proceedings of the sixth ACM international conference on Web search and data mining
Comparative Document Summarization via Discriminative Sentence Selection

ACM Transactions on Knowledge Discovery from Data (TKDD)
Towards content-level coherence with aspect-guided summarization

ACM Transactions on Speech and Language Processing (TSLP)
A pilot study on using profile-based summarisation for interactive search assistance

ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
An integrated semantic-based approach in concept based video retrieval

Multimedia Tools and Applications
A cross-media evolutionary timeline generation framework based on iterative recommendation

Proceedings of the 3rd ACM conference on International conference on multimedia retrieval
Rhetorics-based multi-document summarization

Expert Systems with Applications: An International Journal
Exploring hypergraph-based semi-supervised ranking for query-oriented summarization

Information Sciences: an International Journal
Exploiting relevance, coverage, and novelty for query-focused multi-document summarization

Knowledge-Based Systems
Automatically assessing machine summary content without a gold standard

Computational Linguistics
Sumblr: continuous summarization of evolving tweet streams

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Predicting relevant news events for timeline summaries

Proceedings of the 22nd international conference on World Wide Web companion
Sentence fusion for multidocument news summarization

Computational Linguistics
Gem-based entity-knowledge maintenance

Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
Combining co-clustering with noise detection for theme-based summarization

ACM Transactions on Speech and Language Processing (TSLP)
Multi-document summarization based on the Yago ontology

Expert Systems with Applications: An International Journal
Comments-oriented document summarization based on multi-aspect co-feedback ranking

WAIM'13 Proceedings of the 14th international conference on Web-Age Information Management
Weighted archetypal analysis of the multi-element graph for query-focused multi-document summarization

Expert Systems with Applications: An International Journal
i, poet: automatic Chinese poetry composition through a generative summarization framework under constrained optimization

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
The notion of diversity in graphical entity summarisation on semantic knowledge graphs

Journal of Intelligent Information Systems
Cross-lingual training of summarization systems using annotated corpora in a foreign language

Information Retrieval
Framing image description as a ranking task: data, models and evaluation metrics

Journal of Artificial Intelligence Research
Story graphs: Tracking document set evolution using dynamic graphs

Intelligent Data Analysis - Dynamic Networks and Knowledge Discovery

Quantified Score

Hi-index	0.01

Visualization

Abstract

Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.