Conquering Language: Using NLP on a Massive Scale to Build High Dimensional Language Models from the Web

Authors:
Gregory Grefenstette
Affiliations:
Commissariat à l'Energie Atomique, CEA LIST, SRCI, BP 6, 92265 Fontenay aux Roses Cedex, France
Venue:
CICLing '07 Proceedings of the 8th International Conference on Computational Linguistics and Intelligent Text Processing
Year:
2009

Citing 10
Cited 1

Word association norms, mutual information, and lexicography

Computational Linguistics
Mercator: A scalable, extensible Web crawler

World Wide Web
Measuring praise and criticism: Inference of semantic orientation from association

ACM Transactions on Information Systems (TOIS)
Structural ambiguity and lexical relations

Computational Linguistics - Special issue on using large corpora: I
Identifying, the coding system and language, of on-line documents on the Internet

COLING '96 Proceedings of the 16th conference on Computational linguistics - Volume 2
Learning by googling

ACM SIGKDD Explorations Newsletter
Overview of results of the MUC-6 evaluation

MUC6 '95 Proceedings of the 6th conference on Message understanding
To search or to crawl?: towards a query optimizer for text-centric tasks

Proceedings of the 2006 ACM SIGMOD international conference on Management of data
Finding ideographic representations of Japanese names written in Latin script via language identification and corpus validation

ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
Using the web as an implicit training set: application to structural ambiguity resolution

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing

Large scale semantic structures for image retrieval

Proceedings of the 15th international conference on Multimedia

Quantified Score

Hi-index	0.00

Visualization

Abstract

Dictionaries only contain some of the information we need to know about a language. The growth of the Web, the maturation of linguistic processing tools, and the decline in price of memory storage allow us to envision descriptions of languages that are much larger than before. We can conceive of building a complete language model for a language using all the text that is found on the Web for this language. This article describes our current project to do just that.