Adaptive Retrieval Agents: Internalizing Local Contextand Scaling up to the Web

  • Authors:

  • Filippo Menczer;Richard K. Belew

  • Affiliations:

  • Management Sciences Department, University of Iowa, Iowa City, IA 52242-1000, USA. filippo-menczer@uiowa.edu;Computer Science and Engineering Department, University of California San Diego, La Jolla, CA 92093-0114, USA. rik@cs.ucsd.edu

  • Venue:
  • Machine Learning - Special issue on information retrieval
  • Year:
  • 2000

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper discusses a novel distributed adaptive algorithmand representation used to construct populations of adaptive Webagents. These InfoSpiders browse networked informationenvironments on-line in search of pages relevant to the user, bytraversing hyperlinks in an autonomous and intelligent fashion. Eachagent adapts to the spatial and temporal regularities of its localcontext thanks to a combination of machine learning techniquesinspired by ecological models: evolutionary adaptation with localselection, reinforcement learning and selective query expansion byinternalization of environmental signals, and optional relevancefeedback. We evaluate the feasibility and performance of thesemethods in three domains: a general class of artificial graphenvironments, a controlled subset of the Web, and (preliminarly) thefull Web. Our results suggest that InfoSpiders could take advantageof the starting points provided by search engines, based on globalword statistics, and then use linkage topology to guide their searchon-line. We show how this approach can complement the current stateof the art, especially with respect to the scalability challenge.