Adapting measures of clumping strength to assess term-term similarity

  • Authors:

  • Abraham Bookstein;Vladimir Kulyukin;Timo Raita;John Nicholson

  • Affiliations:

  • Center for Information Studies, University of Chicago, 1010 E, 59 St. Chicago, IL;Computer Science Department, Utah State University, Logan, UT;Computer Science Department, University of Turku, 20520 Turku, Finland;School of Computer Science, DePaul University, 243 S. Wabash Ave. Chicago, IL

  • Venue:
  • Journal of the American Society for Information Science and Technology
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

Automated information retrieval relies heavily on statistical regularities that emerge as terms are deposited to produce text. This paper examines statistical patterns expected of a pair of terms that are semantically related to each other. Guided by a conceptualization of the text generation process, we derive measures of how tightly two terms are semantically associated. Our main objective is to probe whether such measures yield reasonable results. Specifically, we examine how the tendency of a content bearing term to clump, as quantified by previously developed measures of term clumping, is influenced by the presence of other terms. This approach allows us to present a toolkit from which a range of measures can be constructed. As an illustration, one of several suggested measures is evaluated on a large text corpus built from an on-line encyclopedia.