The automatic construction of large-scale corpora for summarization research
Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
Evaluating Natural Language Processing Systems: An Analysis and Review
Evaluating Natural Language Processing Systems: An Analysis and Review
Building a large annotated corpus of English: the penn treebank
Computational Linguistics - Special issue on using large corpora: II
Automatic evaluation of summaries using N-gram co-occurrence statistics
NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Feature-rich part-of-speech tagging with a cyclic dependency network
NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Enriching the knowledge sources used in a maximum entropy part-of-speech tagger
EMNLP '00 Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics - Volume 13
Manual and automatic evaluation of summaries
AS '02 Proceedings of the ACL-02 Workshop on Automatic Summarization - Volume 4
The potential and limitations of automatic sentence extraction for summarization
HLT-NAACL-DUC '03 Proceedings of the HLT-NAACL 03 on Text summarization workshop - Volume 5
The Pyramid Method: Incorporating human content selection variation in summarization evaluation
ACM Transactions on Speech and Language Processing (TSLP)
Measuring importance and query relevance in topic-focused multi-document summarization
ACL '07 Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions
Query-focused summaries or query-biased summaries?
ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
Automatically evaluating content selection in summarization without human models
EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
Text summarisation in progress: a literature review
Artificial Intelligence Review
| Hi-index | 0.00 |
Automated evaluation is crucial in the context of automated text summaries, as is the case with evaluation of any of the language technologies. In this paper we present a Generative Modeling framework for evaluation of content of summaries. We used two simple alternatives to identifying signature-terms from the reference summaries based on model consistency and Parts-Of-Speech (POS) features. By using a Generative Modeling approach we capture the sentence level presence of these signature-terms in peer summaries. We show that parts-of-speech such as noun and verb, give simple and robust method to signature-term identification for the Generative Modeling approach. We also show that having a large set of 'significant signature-terms' is better than a small set of ‘strong signature-terms' for our approach. Our results show that the generative modeling approach is indeed promising — providing high correlations with manual evaluations — and further investigation of signature-term identification methods would obtain further better results. The efficacy of the approach can be seen from its ability to capture ‘overall responsiveness' much better than the state-of-the-art in distinguishing a human from a system.