Bootstrapped language identification for multi-site internet domains

Authors:
Uwe F. Mayer
Affiliations:
eBay Research Labs, San Jose, CA, USA
Venue:
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Year:
2012

Citing 1
Cited 1

Language Identification on the Web: Extending the Dictionary Method

CICLing '09 Proceedings of the 10th International Conference on Computational Linguistics and Intelligent Text Processing

A study in language identification

Proceedings of the Seventeenth Australasian Document Computing Symposium

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present an algorithm for language identification, in particular of short documents, for the case of an Internet domain with sites in multiple countries with differing languages. The algorithm is significantly faster than standard language identification methods, while providing state-of-the-art identification. We bootstrap the algorithm based on the language identification based on the site alone, a methodology suitable for any supervised language identification algorithm. We demonstrate the bootstrapping and algorithm on eBay email data and on Twitter status updates data. The algorithm is deployed at eBay as part of the back-office development data repository.