On automated evaluation of readability of summaries: capturing grammaticality, focus, structure and coherence

Authors:
Ravikiran Vadlapudi;Rahul Katragadda
Affiliations:
Language Technologies Research Center, IIIT Hyderabad;Language Technologies Research Center, IIIT Hyderabad
Venue:
HLT-SRWS '10 Proceedings of the NAACL HLT 2010 Student Research Workshop
Year:
2010

Citing 7
Cited 1

Lexical cohesion computed by thesaural relations as an indicator of the structure of text

Computational Linguistics
TnT: a statistical part-of-speech tagger

ANLC '00 Proceedings of the sixth conference on Applied natural language processing
Modeling local coherence: an entity-based approach

ACL '05 Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics
WikiRelate! computing semantic relatedness using wikipedia

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Predicting the fluency of text with shallow structural features: case studies of machine translation and human-written text

EACL '09 Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics
Revisiting readability: a unified framework for predicting text quality

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Quantitative evaluation of grammaticality of summaries

CICLing'10 Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing

Text summarisation in progress: a literature review

Artificial Intelligence Review

Quantified Score

Hi-index	0.00

Visualization

Abstract

Readability of a summary is usually graded manually on five aspects of readability: grammaticality, coherence and structure, focus, referential clarity and non-redundancy. In the context of automated metrics for evaluation of summary quality, content evaluations have been presented through the last decade and continue to evolve, however a careful examination of readability aspects of summary quality has not been as exhaustive. In this paper we explore alternative evaluation metrics for 'grammaticality' and 'coherence and structure' that are able to strongly correlate with manual ratings. Our results establish that our methods are able to perform pair-wise ranking of summaries based on grammaticality, as strongly as ROUGE is able to distinguish for content evaluations. We observed that none of the five aspects of readability are independent of each other, and hence by addressing the individual criterion of evaluation we aim to achieve automated appreciation of readability of summaries.