Dynamic Provisioning of Virtual Organization Clusters

• Authors: Michael A. Murphy;Brandon Kagey;Michael Fenn;Sebastien Goasguen
• Venue: CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
• Year: 2009

Virtual Organization Clusters are systems comprised of virtual machines that provide dedicated computing clusters for each individual Virtual Organization. The design of these clusters allows individual virtual machines to be independent of the underlying physical hardware, potentially allowing virtual clusters to span multiple grid sites. A major challenge in using Virtual Organization Clusters as a grid computing abstraction arises from the need to schedule and provision physical resources to run the virtual machines.This paper describes a virtual cluster scheduler implementation based on the Condor High Throughput Computing system. By means of real-time monitoring of the Condor job queue, virtual machines that belong to individual Virtual Organizations are provisioned and booted. Jobs belonging to each Virtual Organization are then run on the organization-specific virtual machines, which form a cluster dedicated to the specific organization. Once the queued jobs have executed, the virtual machines are terminated, thereby allowing the physical resources to be re-claimed. Tests of this system were conducted using synthetic workloads, demonstrating that dynamic provisioning of virtual machines preserves system throughput for all but the shortest-running of grid jobs, without undue increase in scheduling latency.