A similarity-based approach to data sparseness problem of chinese language modeling

Authors:
Jinghui Xiao;Bingquan Liu;Xiaolong Wang;Bing Li
Affiliations:
School of Computer Science and Techniques, Harbin Institute of Technology, Harbin, China;School of Computer Science and Techniques, Harbin Institute of Technology, Harbin, China;School of Computer Science and Techniques, Harbin Institute of Technology, Harbin, China;School of Computer Science and Techniques, Harbin Institute of Technology, Harbin, China
Venue:
MICAI'05 Proceedings of the 4th Mexican international conference on Advances in Artificial Intelligence
Year:
2005

Citing 4
Cited 0

Class-based n-gram models of natural language

Computational Linguistics
Similarity-Based Models of Word Cooccurrence Probabilities

Machine Learning - Special issue on natural language learning
The mathematics of statistical machine translation: parameter estimation

Computational Linguistics - Special issue on using large corpora: II
Cooccurrence smoothing for stochastic language modeling

ICASSP'92 Proceedings of the 1992 IEEE international conference on Acoustics, speech and signal processing - Volume 1

Quantified Score

Hi-index	0.00

Visualization

Abstract

Data sparseness problem is inherent and severe in language modeling. Smoothing techniques are usually widely used to solve this problem. However, traditional smoothing techniques are all based on statistical hypotheses without concerning about linguistic knowledge. This paper introduces semantic information into smoothing technique and proposes a similarity-based smoothing method which is based on both statistical hypothesis and linguistic hypothesis. An experiential iterative algorithm is presented to optimize system parameters. Experiment results prove that compared with traditional smoothing techniques, our method can greatly improve the performance of language model.