Short and informal documents: a probabilistic model for description enrichment
NGITS'09 Proceedings of the 7th international conference on Next generation information technologies and systems
| Hi-index | 0.00 |
Chinese and English belong to two very different families of human languages. Yet, since the underlying human concepts are universal, one can expect that there are many statistical similarities between Chinese texts and English texts. In this paper, we present results of analyzing the quantity and frequency of N-grams in 200 million randomly-sampled English and Chinese web pages. The similarities and differences in N-gram frequency distributions yield important insights about the two languages. First, the distribution of the unique number of N-grams is similar between English and Chinese, yet the Chinese distribution is "shifted" to larger N. The distribution indicates that on average, 1.5 Chinese characters correspond to 1 English word. Second, while frequency distributions of uni-grams and bi-grams are very different between Chinese and English, the frequency distribution for 3-grams and 4- grams are strikingly similar between Chinese and English. This leads to the conjecture that in both languages, frequent 3-grams and 4-grams represent the same set of concepts and patterns.