Data compression using dynamic Markov modelling
The Computer Journal
Compression, information theory, and grammars: a unified approach
ACM Transactions on Information Systems (TOIS)
HTML: the definitive guide
The Unicode standard version 3.0
The Unicode standard version 3.0
Introducing Microsoft Windows 2000 Server
Introducing Microsoft Windows 2000 Server
Some Notes on the Formal Properties of Bidirectional Optimality Theory
Journal of Logic, Language and Information
| Hi-index | 0.00 |
The last step of the Information Retrieval process is to display the found documents to the user. However, some difficulties might occur at that point. English texts are usually written in the ASCII standard. Unlike the English language, many languages have different character sets, and do not have one standard. This plurality of standards causes problems, especially in a web environment, where one may download a document with an unknown standard. This paper suggests a purely automatic way of finding the standard which was used by the document writer based on the statistical letters distribution in the language. We developed a vector-space-based method that creates frequencies vectors for each letter of the language and then matches a new document's vectors to the pre-computed templates. The algorithm was applied on various types of corpora in Hebrew, Russian and English, and provides an efficient solution to the stated problem in most cases.