A suite of testbeds for the realistic evaluation of peer-to-peer information retrieval systems

  • Authors:

  • Iraklis A. Klampanos;Victor Poznański;Joemon M. Jose;Peter Dickman

  • Affiliations:

  • University of Glasgow, Glasgow, U.K;Oxford Science Part, Sharp Labs of Europe Ltd., Oxford, U.K;University of Glasgow, Glasgow, U.K;University of Glasgow, Glasgow, U.K

  • Venue:
  • ECIR'05 Proceedings of the 27th European conference on Advances in Information Retrieval Research
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

Peer-to-peer (P2P) networking continuously gains popularity among computing science researchers. The problem of information retrieval (IR) over P2P networks is being addressed by researchers attempting to provide valuable insight as well as solutions for its successful deployment. All published studies have, so far, been evaluated by simulation means, using well-known document collections (usually acquired from TREC). Researchers test their systems using divided collections whose documents have been previously distributed to a number of simulated peers. This practice leads to two problems: First, there is little justification in favour of the document distributions used by relevant studies and second, since different studies use different experimental testbeds, there is no common ground for comparing the solutions proposed. In this work, we contribute a number of different document testbeds for evaluating P2P IR systems. Each of these has been deduced from TREC's WT10g collection and corresponds to different potential P2P IR application scenarios. We analyse each methodology and testbed with respect to the document distributions achieved as well as to the location of relevant items within each setting. This work marks the beginning of an effort to provide more realistic evaluation environments for P2P IR systems as well as to create a common ground for comparisons of existing and future architectures.