A stochastic finite-state word-segmentation algorithm for Chinese

Authors:
Richard Sproat;Chilin Shih;William Gale;Nancy Chang
Affiliations:
AT&T Bell Laboratories, Murray Hill, NJ;AT&T Bell Laboratories, Murray Hill, NJ;AT&T Bell Laboratories, Murray Hill, NJ;Harvard University, Cambridge, MA
Venue:
ACL '94 Proceedings of the 32nd annual meeting on Association for Computational Linguistics
Year:
1994

Citing 6
Cited 23

Chinese text segmentation for text retrieval: achievements and problems

Journal of the American Society for Information Science
A finite-state morphological processor for Spanish

COLING '90 Proceedings of the 13th conference on Computational linguistics - Volume 3
Word identification for Mandarin Chinese sentences

COLING '92 Proceedings of the 14th conference on Computational linguistics - Volume 1
Two-level morphology with composition

COLING '92 Proceedings of the 14th conference on Computational linguistics - Volume 1
Recognizing unregistered names for Mandarin word identification

COLING '92 Proceedings of the 14th conference on Computational linguistics - Volume 4
Weighted rational transductions and their application to human language processing

HLT '94 Proceedings of the workshop on Human Language Technology

On Chinese text retrieval

SIGIR '96 Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval
A statistically emergent approach for language processing: application to modeling context effects in ambiguous Chinese word boundary perception

Computational Linguistics
Overlapping statistical word indexing: a new indexing method for Japanese text

Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval
A Hybrid Approach of Text Segmentation Based on Sensitive Word Concept for NLP

CICLing '01 Proceedings of the Second International Conference on Computational Linguistics and Intelligent Text Processing
An NLP & IR approach to topic detection

Topic detection and tracking
Stochastic inversion transduction grammars and bilingual parsing of parallel corpora

Computational Linguistics
Compound noun segmentation based on lexical data extracted from corpus

Natural Language Engineering
Compound noun segmentation based on lexical data extracted from corpus

ANLC '00 Proceedings of the sixth conference on Applied natural language processing
Improving Chinese tokenization with linguistic filters on statistical lexical acquisition

ANLC '94 Proceedings of the fourth conference on Applied natural language processing
Applying repair processing in Chinese homophone disambiguation

ANLC '97 Proceedings of the fifth conference on Applied natural language processing
CSeg& Tag1.0: a practical word segmenter and POS tagger for Chinese texts

ANLC '97 Proceedings of the fifth conference on Applied natural language processing
Chinese word segmentation without using lexicon and hand-crafted training data

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
An algorithm for simultaneously bracketing parallel texts by aligning words

ACL '95 Proceedings of the 33rd annual meeting on Association for Computational Linguistics
An iterative algorithm to build Chinese language models

ACL '96 Proceedings of the 34th annual meeting on Association for Computational Linguistics
Identification and classification of proper nouns in Chinese texts

COLING '96 Proceedings of the 16th conference on Computational linguistics - Volume 1
Identifying temporal expression and its syntactic role using FST and lexical data from corpus

COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 2
Probabilistic named entity verification

COMPUTERM '02 COLING-02 on COMPUTERM 2002: second international workshop on computational terminology - Volume 14
Backward machine transliteration by learning phonetic similarity

COLING-02 proceedings of the 6th conference on Natural language learning - Volume 20
Applications of corpus-based semantic similarity and word segmentation to database schema matching

The VLDB Journal — The International Journal on Very Large Data Bases
Applying Text Mining to Assist People Who Inquire HIV/AIDS Information from Internet

PAISI, PACCF and SOCO '08 Proceedings of the IEEE ISI 2008 PAISI, PACCF, and SOCO international workshops on Intelligence and Security Informatics
A Joint Segmenting and Labeling Approach for Chinese Lexical Analysis

ECML PKDD '08 Proceedings of the European conference on Machine Learning and Knowledge Discovery in Databases - Part II
A Generalized Approach to Word Segmentation Using Maximum Length Descending Frequency and Entropy Rate

CICLing '07 Proceedings of the 8th International Conference on Computational Linguistics and Intelligent Text Processing
Stochastic inversion transduction grammars with application to segmentation, bracketing, and alignment of parallel corpora

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a stochastic finite-state model for segmenting Chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names. We also evaluate the system's performance, taking into account the fact that people often do not agree on a single segmentation.