Automatic diacritic restoration for resource-scarce languages

Authors:
Guy De Pauw;Peter W. Wagacha;Gilles-Maurice de Schryver
Affiliations:
CNTS - Language Technology Group, University of Antwerp, Belgium;School of Computing and Informatics, University of Nairobi, Kenya;African Languages and Cultures, Ghent University, Belgium and Xhosa Department, University of the Western Cape, South Africa
Venue:
TSD'07 Proceedings of the 10th international conference on Text, speech and dialogue
Year:
2007

Citing 4
Cited 4

DESAM - Annotated Corpus for Czech

SOFSEM '97 Proceedings of the 24th Seminar on Current Trends in Theory and Practice of Informatics: Theory and Practice of Informatics
Diacritics Restoration: Learning from Letters versus Learning from Words

CICLing '02 Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing
Amharic Character Recognition using a Fast Signature Based Algorithm

IV '03 Proceedings of the Seventh International Conference on Information Visualization
Letter level learning for language independent diacritics restoration

COLING-02 proceedings of the 6th conference on Natural language learning - Volume 20

Constrained Sequence Classification for Lexical Disambiguation

PRICAI '08 Proceedings of the 10th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence
The SAWA corpus: a parallel corpus English - Swahili

AfLaT '09 Proceedings of the First Workshop on Language Technologies for African Languages
Diacritics restoration in vietnamese: letter based vs. syllable based model

PRICAI'10 Proceedings of the 11th Pacific Rim international conference on Trends in artificial intelligence
Statistical unicodification of African languages

Language Resources and Evaluation

Quantified Score

Hi-index	0.00

Visualization

Abstract

The orthography of many resource-scarce languages includes diacritically marked characters. Falling outside the scope of the standard Latin encoding, these characters are often represented in digital language resources as their unmarked equivalents. This renders corpus compilation more difficult, as these languages typically do not have the benefit of large electronic dictionaries to perform diacritic restoration. This paper describes experiments with a machine learning approach that is able to automatically restore diacritics on the basis of local graphemic context. We apply the method to the African languages of Cilubà, Gikuyu, Kikamba, Maa, Sesotho sa Leboa, Tshivenda and Yoruba and contrast it with experiments on Czech, Dutch, French, German and Romanian, as well as Vietnamese and Chinese Pinyin.