An investigation of linguistic features and clustering algorithms for topical document clustering

  • Authors:

  • Vasileios Hatzivassiloglou;Luis Gravano;Ankineedu Maganti

  • Affiliations:

  • Department of Computer Science, Columbia Unwersity, 1214 Amsterdam Avenue, New York, NY;Department of Computer Science, Columbia Unwersity, 1214 Amsterdam Avenue, New York, NY;Department of Computer Science, Columbia Unwersity, 1214 Amsterdam Avenue, New York, NY

  • Venue:
  • SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
  • Year:
  • 2000

Quantified Score

Hi-index 0.00

Visualization

Abstract

We investigate four hierarchical clustering methods (single-link, complete-link, groupwise-average, and single-pass) and two linguistically motivated text features (noun phrase heads and proper names) in the context of document clustering. A statistical model for combining similarity information from multiple sources is described and applied to DARPA's Topic Detection and Tracking phase 2 (TDT2) data. This model, based on log-linear regression, alleviates the need for extensive search in order to determine optimal weights for combining input features. Through an extensive series of experiments with more than 40,000 documents from multiple news sources and modalities, we establish that both the choice of clustering algorithm and the introduction of the additional features have an impact on clustering performance. We apply our optimal combination of features to the TDT2 test data, obtaining partitions of the documents that compare favorably with the results obtained by participants in the official TDT2 competition.