Proposing a new term weighting scheme for text categorization

  • Authors:
  • Man Lan;Chew-Lim Tan;Hwee-Boon Low

  • Affiliations:
  • Institute for Infocomm Research, Singapore and School of Computing, National University of Singapore;School of Computing, National University of Singapore, Singapore;Institute for Infocomm Research, Singapore

  • Venue:
  • AAAI'06 Proceedings of the 21st national conference on Artificial intelligence - Volume 1
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

In text categorization, term weighting methods assign appropriate weights to the terms to improve the classification performance. In this study, we propose an effective term weighting scheme, i.e. tf.rf, and investigate several widely-used unsupervised and supervised term weighting methods on two popular data collections in combination with SVM and kNN algorithms. From our controlled experimental results, not all supervised term weighting methods have a consistent superiority over unsupervised term weighting methods. Specifically, the three supervised methods based on the information theory, i.e. tf.χ2, tf.ig and tf.or, perform rather poorly in all experiments. On the other hand, our proposed tf.rf achieves the best performance consistently and outperforms other methods substantially and significantly. The popularly-used tf.idf method has not shown a uniformly good performance with respect to different data corpora.