Integrating local job scheduler – LSFTM with GfarmTM

Authors:
Xiaohui Wei;Wilfred W. Li;Osamu Tatebe;Gaochao Xu;Liang Hu;Jiubin Ju
Affiliations:
College of Computer Science and Technology, Jilin University, Changchun, PRC;University of California, San Diego, CA;Grid Technology Research Center, AIST, Tsukuba, Ibaraki, Japan;College of Computer Science and Technology, Jilin University, Changchun, PRC;College of Computer Science and Technology, Jilin University, Changchun, PRC;College of Computer Science and Technology, Jilin University, Changchun, PRC
Venue:
ISPA'05 Proceedings of the Third international conference on Parallel and Distributed Processing and Applications
Year:
2005

Citing 7
Cited 4

Utopia: a load sharing facility for large, heterogeneous distributed computer systems

Software—Practice & Experience
Condor-G: A Computation Management Agent for Multi-Institutional Grids

Cluster Computing
Preliminary Evaluation of Dynamic Load Balancing Using Loop Re-partitioning on Omni/SCASH

CCGRID '03 Proceedings of the 3st International Symposium on Cluster Computing and the Grid
Managing Network Resources in Condor

HPDC '00 Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing
Decoupling Computation and Data Scheduling in Distributed Data-Intensive Applications

HPDC '02 Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing
Grid Datafarm Architecture for Petascale Data Intensive Computing

CCGRID '02 Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid
A Framework for Self-Optimizing Grids Using P2P Components

DEXA '03 Proceedings of the 14th International Workshop on Database and Expert Systems Applications

Accelerating large-scale data exploration through data diffusion

DADC '08 Proceedings of the 2008 international workshop on Data-aware distributed computing
The quest for scalable support of data-intensive workloads in distributed systems

Proceedings of the 18th ACM international symposium on High performance distributed computing
Making a case for distributed file systems at Exascale

Proceedings of the third international workshop on Large-scale system and application performance
IKAROS: An HTTP-Based Distributed File System, for Low Consumption & Low Specification Devices

Journal of Grid Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Applications that both access and generate large data sets increasingly draw our attention in high energy physics, astronomy, genomics and other disciplines. The Data Grids, like Gfarm, seek to harness geographically distributed resources for such large-scale data-intensive problems. However, scheduling is a challenging task in this context. In this paper, we discuss the integration of LSF with Gfarm. We will discuss how to enable LSF to support Gfarm applications requiring GSI authentication, the design and implementation of data aware scheduling and data management. The system is able to find data-affinity hosts for Gfarm jobs and to adjust the distribution of the data replicas dynamically according to the job load. Before job running, the system will setup the proper credential for it. Using the LSF scheduler plugin mechanism, we do not need to write a new scheduler from scratch or make a lot of changes to an existing scheduler.