A pipelined architecture for distributed text query evaluation

  • Authors:

  • Alistair Moffat;William Webber;Justin Zobel;Ricardo Baeza-Yates

  • Affiliations:

  • Department of Computer Science and Software Engineering, The University of Melbourne, Melbourne, Australia 3010;Department of Computer Science and Software Engineering, The University of Melbourne, Melbourne, Australia 3010 and School of Computer Science and Information Technology, RMIT University, Melbourn ...;School of Computer Science and Information Technology, RMIT University, Melbourne, Australia 3001;Center for Web Research, Department of Computer Science, University of Chile, Santiago, Chile and Yahoo! Research, Barcelona, Spain

  • Venue:
  • Information Retrieval
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

Two principal query-evaluation methodologies have been described for cluster-based implementation of distributed information retrieval systems: document partitioning and term partitioning. In a document-partitioned system, each of the processors hosts a subset of the documents in the collection, and executes every query against its local sub-collection. In a term-partitioned system, each of the processors hosts a subset of the inverted lists that make up the index of the collection, and serves them to a central machine as they are required for query evaluation.In this paper we introduce a pipelined query-evaluation methodology, based on a term-partitioned index, in which partially evaluated queries are passed amongst the set of processors that host the query terms. This arrangement retains the disk read benefits of term partitioning, but more effectively shares the computational load. We compare the three methodologies experimentally, and show that term distribution is inefficient and scales poorly. The new pipelined approach offers efficient memory utilization and efficient use of disk accesses, but suffers from problems with load balancing between nodes. Until these problems are resolved, document partitioning remains the preferred method.