Gathering at the well: creating communities for grid I/O

Authors:
Douglas Thain;John Bent;Andrea Arpaci-Dusseau;Remzi Arpaci-Dusseau;Miron Livny
Affiliations:
University of Wisconsin --- Madison, Madison, WI;University of Wisconsin --- Madison, Madison, WI;University of Wisconsin --- Madison, Madison, WI;University of Wisconsin --- Madison, Madison, WI;University of Wisconsin --- Madison, Madison, WI
Venue:
Proceedings of the 2001 ACM/IEEE conference on Supercomputing
Year:
2001

Citing 19
Cited 14

Scale and performance in a distributed file system

ACM Transactions on Computer Systems (TOCS)
Interposition agents: transparently interposing user code at the system interface

SOSP '93 Proceedings of the fourteenth ACM symposium on Operating systems principles
UFO: a personal global file system based on user-level extensions to the operating system

ACM Transactions on Computer Systems (TOCS)
GASS: a data movement and access service for wide area computing systems

Proceedings of the sixth workshop on I/O in parallel and distributed systems
The Jini architecture for network-centric computing

Communications of the ACM
Implementing remote procedure calls

ACM Transactions on Computer Systems (TOCS)
A survey of web caching schemes for the Internet

ACM SIGCOMM Computer Communication Review
Multiple Bypass: Interposition Agents for Distributed Computing

Cluster Computing
The SDSC storage resource broker

CASCON '98 Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research
Replica Selection in the Globus Data Grid

CCGRID '01 Proceedings of the 1st International Symposium on Cluster Computing and the Grid
A Case for Buffer Servers

HOTOS '99 Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems
Matchmaking: Distributed Resource Management for High Throughput Computing

HPDC '98 Proceedings of the 7th IEEE International Symposium on High Performance Distributed Computing
Process Hijacking

HPDC '99 Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing
Resource Management through Multilateral Matchmaking

HPDC '00 Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing
Grid-Based File Access: The Legion I/O Model

HPDC '00 Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing
Bypass: A Tool for Building Split Execution Systems

HPDC '00 Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing
The Kangaroo Approach to Data Movement on the Grid

HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing
Matchmaking frameworks for distributed resource management

Matchmaking frameworks for distributed resource management
Flash: an efficient and portable web server

ATEC '99 Proceedings of the annual conference on USENIX Annual Technical Conference

Decoupling Computation and Data Scheduling in Distributed Data-Intensive Applications

HPDC '02 Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing
Flexibility, Manageability, and Performance in a Grid Storage Appliance

HPDC '02 Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing
References

Grid resource management
A taxonomy of Data Grids for distributed data sharing, management, and processing

ACM Computing Surveys (CSUR)
An adaptive meta-scheduler for data-intensive applications

International Journal of Grid and Utility Computing
Adaptive hierarchical scheduling policy for enterprise grid computing systems

Journal of Network and Computer Applications
Scalable and distributed mechanisms for integrated scheduling and replication in data grids

ICDCN'08 Proceedings of the 9th international conference on Distributed computing and networking
Discovering Piecewise Linear Models of Grid Workload

CCGRID '10 Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
Towards Non-Stationary Grid Models

Journal of Grid Computing
Evolving toward the perfect schedule: co-scheduling job assignments and data replication in wide-area systems using a genetic algorithm

JSSPP'05 Proceedings of the 11th international conference on Job Scheduling Strategies for Parallel Processing
Integration of scheduling and replication in data grids

HiPC'04 Proceedings of the 11th international conference on High Performance Computing
Data driven infrastructure and policy selection to enhance scientific applications in grid

SAG'04 Proceedings of the First international conference on Scientific Applications of Grid Computing
A Bee Colony based optimization approach for simultaneous job scheduling and data replication in grid environments

Computers and Operations Research
Hopfield neural network for simultaneous job scheduling and data replication in grids

Future Generation Computer Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

Grid applications have demanding I/O needs. Schedulers must bring jobs and data in close proximity in order to satisfy throughput, scalability, and policy requirements. Most systems accomplish this by making either jobs or data mobile. We propose a system that allows jobs and data to meet by binding execution and storage sites together into I/O communities which then participate in the wide-area system. The relationships between participants in a community may be expressed by the ClassAd framework. Extensions to the framework allow community members to express indirect relations. We demonstrate our implementation of I/O communities by improving the performance of a key high-energy physics simulation on an international distributed system.