Scheduling of tasks with batch-shared I/O on heterogeneous systems

  • Authors:

  • Nagavijayalakshmi Vydyanathan;Gaurav Khanna;Umit Catalyurek;Tahsin Kurc;P. Sadayappan;Joel Saltz

  • Affiliations:

  • Dept. of Computer Science and Engineering, The Ohio State University, Columbus, OH;Dept. of Computer Science and Engineering, The Ohio State University, Columbus, OH;Dept. of Biomedical Informatics, The Ohio State University, Columbus, OH and Dept. of Electrical and Computer Engineering, The Ohio State University, Columbus, OH;Dept. of Biomedical Informatics, The Ohio State University, Columbus, OH;Dept. of Computer Science and Engineering, The Ohio State University, Columbus, OH;Dept. of Computer Science and Engineering, The Ohio State University, Columbus, OH and Dept. of Biomedical Informatics, The Ohio State University, Columbus, OH

  • Venue:
  • IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper proposes a novel strategy that uses hypergraph partitioning and K-way iterative mapping-refinement heuristics for scheduling a batch of data-intensive tasks with batch-shared I/O behavior on heterogeneous collections of storage and compute clusters. The strategy formulates file sharing among tasks as a hypergraph to minimize the I/O overheads due to duplicate file transfers and employs a K-way iterative mapping-refinement scheme to adapt to the heterogeneity of compute clusters and storage networks in the system. We evaluate the proposed approach through real experiments and simulations on application scenarios from two application domains; satellite data processing and biomedical imaging. Our experimental results show that our approach can achieve significant performance improvement over algorithms such as HPS, Shortest Job First, MinMin, MaxMin and Sufferage for workloads with high degree of shared I/O among tasks.