Non-word identification or spell checking without a dictionary

  • Authors:
  • Donald C. Comeau;W. John Wilbur

  • Affiliations:
  • National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Building 38A, Room N611S, 8600 Rockville Pike, Bethesda, MD;National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Building 38A, Room N611S, 8600 Rockville Pike, Bethesda, MD

  • Venue:
  • Journal of the American Society for Information Science and Technology
  • Year:
  • 2004

Quantified Score

Hi-index 0.00

Visualization

Abstract

MEDLINE® is a collection of more than 12 million references and abstracts covering recent life science literature. With its continued growth and cutting-edge terminology, spell-checking with a traditional lexicon based approach requires significant additional manual followup. In this work, an internal corpus based context quality rating α, frequency, and simple misspelling transformations are used to rank words from most likely to be misspellings to least likely. Eleven-point average precisions of 0.891 have been achieved within a class of 42,340 all alphabetic words having an α score less than 10. Our models predict that 16,274 or 38% of these words are misspellings. Based on test data, this result has a recall of 79% and a precision of 86%. In other words, spell checking can be done by statistics instead of with a dictionary. As an application we examine the time history of low α words in MEDLINE® titles and abstracts.