Decentralized load balancing for highly irregular search problems

  • Authors:
  • Giuseppe Di Fatta;Michael R. Berthold

  • Affiliations:
  • School of Systems Engineering, The University of Reading, Whiteknights, Reading, Berkshire, RG6 6AY, UK;Department of Computer and Information Science, University of Konstanz, Box M712, 78457 Konstanz, Germany

  • Venue:
  • Microprocessors & Microsystems
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

In this paper, we present a distributed computing framework for problems characterized by a highly irregular search tree, whereby no reliable workload prediction is available. The framework is based on a peer-to-peer computing environment and dynamic load balancing. The system allows for dynamic resource aggregation, does not depend on any specific meta-computing middleware and is suitable for large-scale, multi-domain, heterogeneous environments, such as computational Grids. Dynamic load balancing policies based on global statistics are known to provide optimal load balancing performance, while randomized techniques provide high scalability. The proposed method combines both advantages and adopts distributed job-pools and a randomized polling technique. The framework has been successfully adopted in a parallel search algorithm for subgraph mining and evaluated on a molecular compounds dataset. The parallel application has shown good scalability and close-to linear speedup in a distributed network of workstations.