Automatic topic detection with an incremental clustering algorithm

  • Authors:
  • Xiaoming Zhang;Zhoujun Li

  • Affiliations:
  • School of computer, Beihang University, Beijng, China;School of computer, Beihang University, Beijng, China

  • Venue:
  • WISM'10 Proceedings of the 2010 international conference on Web information systems and mining
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

At present, most of the topic detection approaches are not accurate and efficient enough. In this paper, we proposed a new topic detection method (TPIC) based on an incremental clustering algorithm. It employs a self-refinement process of discriminative feature identification and a term reweighting algorithm to accurately cluster the given documents which discuss the same topic. To be efficient, the "aging" nature of topics is used to precluster stories. To automatically detect the true number of topics, Bayesian Information Criterion (BIC) is used to estimate the true number of topics. Experimental results on Linguistic Data Consortium (LDC) datasets TDT4 show that the proposed method can improve both the efficiency and accuracy, compared to other methods.