Thai Word Segmentation with Hidden Markov Model and Decision Tree

Authors:
Poramin Bheganan;Richi Nayak;Yue Xu
Affiliations:
Faculty of Science and Technology, Queensland University of Technology, Australia;Faculty of Science and Technology, Queensland University of Technology, Australia;Faculty of Science and Technology, Queensland University of Technology, Australia
Venue:
PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
Year:
2009

Citing 4
Cited 1

C4.5: programs for machine learning

C4.5: programs for machine learning
Automatic corpus-based Thai word extraction with the c4.5 learning algorithm

COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 2
Context-based spelling correction for Japanese OCR

COLING '96 Proceedings of the 16th conference on Computational linguistics - Volume 2
Non-dictionary-based Thai word segmentation using decision trees

HLT '01 Proceedings of the first international conference on Human language technology research

Multilingual OCR research and applications: an overview

Proceedings of the 4th International Workshop on Multilingual OCR

Quantified Score

Hi-index	0.00

Visualization

Abstract

The Thai written language is one of the languages that does not have word boundaries. In order to discover the meaning of the document, all texts must be separated into syllables, words, sentences, and paragraphs. This paper develops a novel method to segment the Thai text by combining a non-dictionary based technique with a dictionary-based technique. This method first applies the Thai language grammar rules to the text for identifying syllables. The hidden Markov model is then used for merging possible syllables into words. The identified words are verified with a lexical dictionary and a decision tree is employed to discover the words unidentified by the lexical dictionary. Documents used in the litigation process of Thai court proceedings have been used in experiments. The results which are segmented words, obtained by the proposed method outperform the results obtained by other existing methods.