Using the omega index for evaluating abstractive community detection

Authors:
Gabriel Murray;Giuseppe Carenini;Raymond Ng
Affiliations:
University of the Fraser Valley;University of British Columbia;University of British Columbia
Venue:
Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization
Year:
2012

Citing 12
Cited 1

The use of MMR, diversity-based reranking for reordering documents and producing summaries

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
New Methods in Automatic Extracting

Journal of the ACM (JACM)
The TIPSTER SUMMAC Text Summarization Evaluation

EACL '99 Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics
Sentence Fusion for Multidocument News Summarization

Computational Linguistics
Speech and Language Processing (2nd Edition)

Speech and Language Processing (2nd Edition)
Identifying agreement and disagreement in conversational speech: use of Bayesian networks to model pragmatic dependencies

ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
Minimum cut model for spoken lecture segmentation

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
An Algorithm to Find Overlapping Community Structure in Networks

PKDD 2007 Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases
A Fast Algorithm to Find Overlapping Communities in Networks

ECML PKDD '08 Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases - Part I
The automatic creation of literature abstracts

IBM Journal of Research and Development
Exploiting conversation structure in unsupervised topic segmentation for emails

EMNLP '10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing
Generating and validating abstracts of meeting conversations: a user study

INLG '10 Proceedings of the 6th International Natural Language Generation Conference

Scalable parallel OPTICS data clustering using graph algorithmic techniques

SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Quantified Score

Hi-index	0.00

Visualization

Abstract

Numerous NLP tasks rely on clustering or community detection algorithms. For many of these tasks, the solutions are disjoint, and the relevant evaluation metrics assume nonoverlapping clusters. In contrast, the relatively recent task of abstractive community detection (ACD) results in overlapping clusters of sentences. ACD is a sub-task of an abstractive summarization system and represents a twostep process. In the first step, we classify sentence pairs according to whether the sentences should be realized by a common abstractive sentence. This results in an undirected graph with sentences as nodes and predicted abstractive links as edges. The second step is to identify communities within the graph, where each community corresponds to an abstractive sentence to be generated. In this paper, we describe how the Omega Index, a metric for comparing non-disjoint clustering solutions, can be used as a summarization evaluation metric for this task. We use the Omega Index to compare and contrast several community detection algorithms.