Evaluating Natural Language Processing Systems: An Analysis and Review
Evaluating Natural Language Processing Systems: An Analysis and Review
SUMMAC: a text summarization evaluation
Natural Language Engineering
Meta-evaluation of summaries in a cross-lingual environment using content-based metrics
COLING '02 Proceedings of the 19th international conference on Computational linguistics - Volume 1
BLEU: a method for automatic evaluation of machine translation
ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Automatic evaluation of summaries using N-gram co-occurrence statistics
NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Evaluation challenges in large-scale document summarization
ACL '03 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1
An information-theoretic approach to automatic evaluation of summaries
HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
Information Processing and Management: an International Journal
A comparison of rankings produced by summarization evaluation measures
NAACL-ANLP-AutoSum '00 Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization
Evalinitiatives '03 Proceedings of the EACL 2003 Workshop on Evaluation Initiatives in Natural Language Processing: are evaluation methods, metrics and resources reusable?
Automatically evaluating content selection in summarization without human models
EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
Evaluation of automatic summaries: metrics under varying data conditions
UCNLG+Sum '09 Proceedings of the 2009 Workshop on Language Generation and Summarisation
MICAI'07 Proceedings of the artificial intelligence 6th Mexican international conference on Advances in artificial intelligence
The GIL summarizers: experiments in the track QA@INEX'10
INEX'10 Proceedings of the 9th international conference on Initiative for the evaluation of XML retrieval: comparative evaluation of focused retrieval
The REG summarization system with question reformulation at QA@INEX track 2010
INEX'10 Proceedings of the 9th international conference on Initiative for the evaluation of XML retrieval: comparative evaluation of focused retrieval
ACM SIGIR Forum
Resolving ambiguity in biomedical text to improve summarization
Information Processing and Management: an International Journal
Summarizing speech by contextual reinforcement of important passages
PROPOR'12 Proceedings of the 10th international conference on Computational Processing of the Portuguese Language
Machine translation for multilingual summary content evaluation
Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization
Summary evaluation: together we stand NPowER-ed
CICLing'13 Proceedings of the 14th international conference on Computational Linguistics and Intelligent Text Processing - Volume 2
MICAI'12 Proceedings of the 11th Mexican international conference on Advances in Computational Intelligence - Volume Part II
Automatically assessing machine summary content without a gold standard
Computational Linguistics
| Hi-index | 0.00 |
We study correlation of rankings of text summarization systems using evaluation methods with and without human models. We apply our comparison framework to various well-established content-based evaluation measures in text summarization such as coverage, Responsiveness, Pyramids and Rouge studying their associations in various text summarization tasks including generic and focus-based multi-document summarization in English and generic single-document summarization in French and Spanish. The research is carried out using a new content-based evaluation framework called Fresa to compute a variety of divergences among probability distributions.