Automatic evaluation of text coherence: models and representations

Authors:
Mirella Lapata;Regina Barzilay
Affiliations:
School of Informatics, University of Edinburgh, Edinburgh, UK;Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA
Venue:
IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Year:
2005

Citing 12
Cited 30

Computer systems that learn: classification and prediction methods from statistics, neural nets, machine learning, and expert systems

Computer systems that learn: classification and prediction methods from statistics, neural nets, machine learning, and expert systems
Centering: a framework for modeling the local coherence of discourse

Computational Linguistics
Using linear algebra for intelligent information retrieval

SIAM Review
Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone

SIGDOC '86 Proceedings of the 5th annual international conference on Systems documentation
An Information-Theoretic Definition of Similarity

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Head-driven statistical models for natural language parsing

Head-driven statistical models for natural language parsing
Lexical cohesion computed by thesaural relations as an indicator of the structure of text

Computational Linguistics
Evaluation of text coherence for electronic essay scoring systems

Natural Language Engineering
BLEU: a method for automatic evaluation of machine translation

ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Automatic evaluation of summaries using N-gram co-occurrence statistics

NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Evaluation metrics for generation

INLG '00 Proceedings of the first international conference on Natural language generation - Volume 14
Using information content to evaluate semantic similarity in a taxonomy

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 1

Automatic summarising: The state of the art

Information Processing and Management: an International Journal
Categorizing and ranking search engine's results by semantic similarity

Proceedings of the 2nd international conference on Ubiquitous information management and communication
Semantic text similarity using corpus-based word similarity and string similarity

ACM Transactions on Knowledge Discovery from Data (TKDD)
Semantically driven snippet selection for supporting focused web searches

Data & Knowledge Engineering
Ontology-based relevance analysis for automatic reference tracking

International Journal of Computer Applications in Technology
Coreference-inspired coherence modeling

HLT-Short '08 Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers
Corpus-based and knowledge-based measures of text semantic similarity

AAAI'06 Proceedings of the 21st national conference on Artificial intelligence - Volume 1
Extending the entity-grid coherence model to semantically related entities

ENLG '07 Proceedings of the Eleventh European Workshop on Natural Language Generation
Automatically evaluating content selection in summarization without human models

EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
Processing methods for contextual coherence interpretation related to reputation

ISCIT'09 Proceedings of the 9th international conference on Communications and information technologies
Using complexity measures in information retrieval

Proceedings of the third symposium on Information interaction in context
Automatic evaluation of linguistic quality in multi-document summarization

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Structural features for predicting the linguistic quality of text: applications to machine translation, automatic summarization and human-authored text

Empirical methods in natural language generation
Comparative evaluation of ontology-based Automatic Reference Tracking (ART)

International Journal of Networking and Virtual Organisations
A comparison of features for automatic readability assessment

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Word sense disambiguation-based sentence similarity

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Automatically evaluating text coherence using discourse relations

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Disentangling chat with local coherence models

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Extending the entity grid with entity-specific features

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2
Text summarisation in progress: a literature review

Artificial Intelligence Review
Quantitative evaluation of grammaticality of summaries

CICLing'10 Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing
Resolving ambiguity in biomedical text to improve summarization

Information Processing and Management: an International Journal
Supporting the review of student proposal drafts in information technologies

Proceedings of the 13th annual conference on Information technology education
A simple unsupervised latent semantics based approach for sentence similarity

SemEval '12 Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation
UNT: a supervised synergistic approach to semantic text similarity

SemEval '12 Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation
Modeling sentences in the latent space

ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1
A coherence model based on syntactic patterns

EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning
Reducing information redundancy in search results

Proceedings of the 28th Annual ACM Symposium on Applied Computing
Synthetic review spamming and defense

Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
The emergence of semantic meaning in the ventral temporal pathway

Journal of Cognitive Neuroscience

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper investigates the automatic evaluation of text coherence for machine-generated texts. We introduce a fully-automatic, linguistically rich model of local coherence that correlates with human judgments. Our modeling approach relies on shallow text properties and is relatively inexpensive. We present experimental results that assess the predictive power of various discourse representations proposed in the linguistic literature. Our results demonstrate that certain models capture complementary aspects of coherence and thus can be combined to improve performance.