Topic Distillation and Spectral Filtering

  • Authors:
  • Soumen Chakrabarti;Byron E. Dom;David Gibson;Ravi Kumar;Prabhakar Raghavan;Sridhar Rajagopalan;Andrew Tomkins

  • Affiliations:
  • IBM Research Division, Almaden Research Center, 650 Harry Rd., San Jose, CA 95120-6099, USA/ soumen@almaden.ibm.com;IBM Research Division, Almaden Research Center, 650 Harry Rd., San Jose, CA 95120-6099, USA/ dom@almaden.ibm.com;IBM Research Division, Almaden Research Center, 650 Harry Rd., San Jose, CA 95120-6099, USA/ gibson@almaden.ibm.com;IBM Research Division, Almaden Research Center, 650 Harry Rd., San Jose, CA 95120-6099, USA/ ravi@almaden.ibm.com;IBM Research Division, Almaden Research Center, 650 Harry Rd., San Jose, CA 95120-6099, USA/ pragh@almaden.ibm.com;IBM Research Division, Almaden Research Center, 650 Harry Rd., San Jose, CA 95120-6099, USA/ sridhar@almaden.ibm.com;IBM Research Division, Almaden Research Center, 650 Harry Rd., San Jose, CA 95120-6099, USA

  • Venue:
  • Artificial Intelligence Review - Special issue on data mining on the Internet
  • Year:
  • 1999

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper discuss topic distillation, an information retrieval problemthat is emerging as a critical task for the www. Algorithms for this problemmust distill a small number of high-quality documents addressing a broadtopic from a large set of candidates.We give a review of the literature, and compare the problem with relatedtasks such as classification, clustering, and indexing. We then describe ageneral approach to topic distillation with applications to searching andpartitioning, based on the algebraic properties of matrices derived fromparticular documents within the corpus. Our method – which we call special filtering – combines the use of terms, hyperlinks and anchor-textto improve retrieval performance. We give results for broad-topic querieson the www, and also give some anecdotal results applying the sametechniques to US Supreme Court law cases, US patents, and a set of WallStreet Journal newspaper articles.