Sync/Async parallel search for the efficient design and construction of web search engines

  • Authors:
  • Mauricio Marin;Veronica Gil-Costa;Carolina Bonacic;Ricardo Baeza-Yates;Isaac D. Scherson

  • Affiliations:
  • Yahoo! Research Latin America, Santiago, Chile and Informatic Engineering Department, University of Santiago of Chile, Chile;Yahoo! Research Latin America, Santiago, Chile and Informatic Department, National University of San Luis, Argentina;Computer Architecture Department, Complutense University of Madrid, Spain;Yahoo! Research Latin America, Santiago, Chile;Department of Computer Science, University of California, Irvine, CA 92697, United States

  • Venue:
  • Parallel Computing
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

A parallel query processing method is proposed for the design and construction of web search engines to efficiently deal with dynamic variations in query traffic. The method allows for the efficient use of different distributed indexing and query processing strategies in server clusters consisting of multiple computational/storage nodes. It also enables a better utilization of local and distributed hardware resources as it automatically re-organizes parallel computations to benefit from the advantages of two mixed modes of operation, namely: a newly proposed synchronous mode and the standard asynchronous computing mode. Switching between modes is facilitated by a round-robin strategy devised to grant each query a fair share of the hardware resources and properly predict query throughput. Performance is evaluated by experimental methods and two case studies serve to show how to develop efficient parallel query processing algorithms for large-scale search engines based on the proposed paradigm.