Free-gram phrase identification for modeling Chinese text

  • Authors:
  • Xi Peng;Zhang Yi;Xiao-Yong Wei;De-Zhong Peng;Yong-Sheng Sang

  • Affiliations:
  • Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, 610065, China;Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, 610065, China;Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, 610065, China;Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, 610065, China;Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, 610065, China

  • Venue:
  • Information Processing Letters
  • Year:
  • 2013

Quantified Score

Hi-index 0.89

Visualization

Abstract

Vector space model using bag of phrases plays an important role in modeling Chinese text. However, the conventional way of using fixed gram scanning to identify free-length phrases is costly. To address this problem, we propose a novel approach for key phrase identification which is capable of identify phrases with all lengths and thus improves the coding efficiency and discrimination of the data representation. In the proposed method, we first convert each document into a context graph, a directed graph that encapsulates the statistical and positional information of all the 2-word strings in the document. We treat every transmission path in the graph as a hypothesis for a phrase, and select the corresponding phrase as a candidate phrase if the hypothesis is valid in the original document. Finally, we selectively divide some of the complex candidate phrases into sub-phrases to improve the coding efficiency, resulting in a set of phrases for codebook construction. The experiments on both balanced and unbalanced datasets show that the codebooks generated by our approach are more efficient than those by conventional methods (one syntactical method and three statistical methods are investigated). Furthermore, the data representation created by our approach has demonstrated higher discrimination than those by conventional methods in classification task.