Condor-G: A Computation Management Agent for Multi-Institutional Grids

Authors:
James Frey;Todd Tannenbaum;Miron Livny;Ian Foster;Steven Tuecke
Affiliations:
Department of Computer Science, University of Wisconsin, Madison, WI 53706, USA;Department of Computer Science, University of Wisconsin, Madison, WI 53706, USA;Department of Computer Science, University of Wisconsin, Madison, WI 53706, USA;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL 60439, USA;Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL 60439, USA
Venue:
Cluster Computing
Year:
2002

Citing 21
Cited 170

A worldwide flock of Condors: load sharing among workstation clusters

Future Generation Computer Systems - Special issue: resource management in distributed systems
From the I-WAY to the National Technology Grid

Communications of the ACM
A security architecture for computational grids

CCS '98 Proceedings of the 5th ACM conference on Computer and communications security
The Globus toolkit

The grid
High-performance schedulers

The grid
High-throughput resource management

The grid
GASS: a data movement and access service for wide area computing systems

Proceedings of the sixth workshop on I/O in parallel and distributed systems
Application-level scheduling on distributed heterogeneous networks

Supercomputing '96 Proceedings of the 1996 ACM/IEEE conference on Supercomputing
The AppLeS parameter sweep template: user-level middleware for the grid

Proceedings of the 2000 ACM/IEEE conference on Supercomputing
SETI@HOME—massively distributed computing for SETI

Computing in Science and Engineering
Comparing Job-Management Systems: The User's Perspective

IEEE Computational Science & Engineering
A Resource Management Architecture for Metacomputing Systems

IPPS/SPDP '98 Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing
Replica Selection in the Globus Data Grid

CCGRID '01 Proceedings of the 1st International Symposium on Cluster Computing and the Grid
Nimrod: a tool for performing parametrised simulations using distributed workstations

HPDC '95 Proceedings of the 4th IEEE International Symposium on High Performance Distributed Computing
Grids as Production Computing Environments: The Engineering Aspects of NASA's Information Power Grid

HPDC '99 Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing
Resource Management through Multilateral Matchmaking

HPDC '00 Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing
Constructing the ASCI Computational Grid

HPDC '00 Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing
Grid Information Services for Distributed Resource Sharing

HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing
An Online Credential Repository for the Grid: MyProxy

HPDC '01 Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing
Solving large-scale quadratic assignment problems

Solving large-scale quadratic assignment problems
The Anatomy of the Grid: Enabling Scalable Virtual Organizations

International Journal of High Performance Computing Applications

A decoupled scheduling approach for Grid application development environments

Journal of Parallel and Distributed Computing - Special issue on computational grids
Service-Level Agreements and Commercial Grids

IEEE Internet Computing
References

Grid resource management
Building with paradisEO reusable parallel and distributed evolutionary algorithms

Parallel Computing - Special issue: Parallel and nature-inspired computational paradigms and applications
Application partitioning and hierarchical management in grid environments

DSM '04 Proceedings of the 1st international doctoral symposium on Middleware
Grid middleware services for virtual data discovery, composition, and integration

MGC '04 Proceedings of the 2nd workshop on Middleware for grid computing
Competitive proportional resource allocation policy for computational grid

Future Generation Computer Systems - Special issue: Computational science of lattice Boltzmann modelling
A framework for adaptive execution in grids

Software—Practice & Experience
Self-Organizing Agents for Grid Load Balancing

GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
High-Level Grid Application Environment to Use Legacy Codes as OGSA Grid Services

GRID '04 Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing
Resource Management for Rapid Application Turnaround on Enterprise Desktop Grids

Proceedings of the 2004 ACM/IEEE conference on Supercomputing
GIPSE: Streamlining the Management of Simulation on the Grid

ANSS '05 Proceedings of the 38th annual Symposium on Simulation
NetSolve/D: A Massively Parallel Grid Execution System for Scalable Data Intensive Collaboration

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 10 - Volume 11
Avalanche Photodiode Gain and Impulse Response Calculation on a Public Computing Platform

IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 13 - Volume 14
Grid load balancing using intelligent agents

Future Generation Computer Systems
The Anatomy of the Grid: Enabling Scalable Virtual Organizations

International Journal of High Performance Computing Applications
Simplifying construction of complex workflows for non-expert users of the Southern California Earthquake Center Community Modeling Environment

ACM SIGMOD Record
JOIN: the implementation of a Java-based massively parallel grid

Future Generation Computer Systems - Special issue: Parallel computing technologies
Utility driven dynamic resource allocation using competitive markets in computational grid

Advances in Engineering Software
Building and Managing the eMinerals Clusters: A Case Study in Grid-Enabled Cluster Operation

Computing in Science and Engineering
Hierarchical submission in a Grid environment

MGC '05 Proceedings of the 3rd international workshop on Middleware for grid computing
Grid Enablement and Performance Evaluation for Simulators of Nanoscale Measurements of Material Properties

HPCASIA '05 Proceedings of the Eighth International Conference on High-Performance Computing in Asia-Pacific Region
Workload management of cooperatively federated computing clusters

The Journal of Supercomputing
Scheduling of a meta-task with QoS requirements in heterogeneous computing systems

Journal of Parallel and Distributed Computing
PGGA: a predictable and grouped genetic algorithm for job scheduling

Future Generation Computer Systems - Parallel input/output management techniques (PIOMT) in cluster and grid computing
Resource management and organization in CROWN grid

InfoScale '06 Proceedings of the 1st international conference on Scalable information systems
Distributed data storage for opportunistic grids

Proceedings of the 3rd international Middleware doctoral symposium
Efficient task replication and management for adaptive fault tolerance in mobile Grid environments

Future Generation Computer Systems - Special section: Information engineering and enterprise architecture in distributed computing environments
Advanced resource connector middleware for lightweight computational Grids

Future Generation Computer Systems - Special section: Information engineering and enterprise architecture in distributed computing environments
A modular meta-scheduling architecture for interfacing with pre-WS and WS Grid resource management services

Future Generation Computer Systems - Special section: Information engineering and enterprise architecture in distributed computing environments
Evaluation of service selection techniques in service oriented computing networks

Multiagent and Grid Systems - Smart Grid Technologies & Market Models
Pegasus: A framework for mapping complex scientific workflows onto distributed systems

Scientific Programming
Worldwide computing: Adaptive middleware and programming technology for dynamic Grid environments

Scientific Programming - Dynamic Grids and Worldwide Computing
Virtual workspaces: Achieving quality of service and quality of life in the Grid

Scientific Programming - Dynamic Grids and Worldwide Computing
Development and execution of an impact cratering application on a computational Grid

Scientific Programming
A low-cost rescheduling policy for efficient mapping of workflows on grid systems

Scientific Programming - AxGrids 2004
S-Club: an overlay-based efficient service discovery mechanism in CROWN Grid

Knowledge and Information Systems
Workflow automation for processing plasma fusion simulation data

Proceedings of the 2nd workshop on Workflows in support of large-scale science
The Internet Operating System: Middleware for Adaptive Distributed Computing

International Journal of High Performance Computing Applications
Fair Scheduling Algorithms in Grids

IEEE Transactions on Parallel and Distributed Systems
Towards self-organising agent-based resource allocation in a multi-server environment

Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems
Allocation strategies for utilization of space-shared resources in Bag of Tasks grids

Future Generation Computer Systems
Grid query optimizer to improve query processing in grids

Future Generation Computer Systems
Falkon: a Fast and Light-weight tasK executiON framework

Proceedings of the 2007 ACM/IEEE conference on Supercomputing
A user-level secure grid file system

Proceedings of the 2007 ACM/IEEE conference on Supercomputing
A high-throughput application for the dynamic analysis of structures on a Grid environment

Advances in Engineering Software
A comparison between two grid scheduling philosophies: EGEE WMS and Grid Way

Multiagent and Grid Systems - Grid Computing, high performance and distributed applications
Combining batch execution and leasing using virtual machines

HPDC '08 Proceedings of the 17th international symposium on High performance distributed computing
Designing a resource broker for heterogeneous grids

Software—Practice & Experience
A probabilistic and adaptive scheduling algorithm using system-generated predictions for inter-grid resource sharing

The Journal of Supercomputing
Parallel computing and the Grid-experiences and applications

International Journal of Parallel, Emergent and Distributed Systems
Toward loosely coupled programming on petascale systems

Proceedings of the 2008 ACM/IEEE conference on Supercomputing
Grid Services for Parallel Molecular Dynamics with NAMD and CHARMM

ICCSA '08 Proceeding sof the international conference on Computational Science and Its Applications, Part I
Grid-Supported Simulation of Vapour-Liquid Equilibria with GridSFEA

ICCS '08 Proceedings of the 8th international conference on Computational Science, Part I
QoS-Oriented Reputation-Aware Query Scheduling in Data Grids

Euro-Par '08 Proceedings of the 14th international Euro-Par conference on Parallel Processing
Flying Low: Simple Leases with Workspace Pilot

Euro-Par '08 Proceedings of the 14th international Euro-Par conference on Parallel Processing
On the use of meta-heuristics to increase the efficiency of online grid workflow scheduling algorithms

Cluster Computing
A recursive architecture for hierarchical grid resource management

Future Generation Computer Systems
A workflow model for heterogeneous computing environments

Future Generation Computer Systems
The MarineGrid project in Ireland with Webcom

Computers & Geosciences
On the design of communication-aware fault-tolerant scheduling algorithms for precedence constrained tasks in grid computing systems with dedicated communication devices

Journal of Parallel and Distributed Computing
Data placement for scientific applications in distributed environments

GRID '07 Proceedings of the 8th IEEE/ACM International Conference on Grid Computing
Supporting adaptable applications in grid resource management systems

GRID '07 Proceedings of the 8th IEEE/ACM International Conference on Grid Computing
A semantic GRID resource matchmaker

ICCOMP'08 Proceedings of the 12th WSEAS international conference on Computers
A sampling-based scheduling method for distributed computing

CISST'09 Proceedings of the 3rd WSEAS international conference on Circuits, systems, signal and telecommunications
Efficient maintenance of distributed data in highly dynamic opportunistic grids

Proceedings of the 2009 ACM symposium on Applied Computing
A sampling-based method for dynamic scheduling in distributed data mining environment

WSEAS Transactions on Computers
Service-oriented grid computing system for digital rights management (GC-DRM)

Expert Systems with Applications: An International Journal
Service scheduling and rescheduling in an applications integration framework

Advances in Engineering Software
A new grid resource management mechanism with resource-aware policy administrator for SLA-constrained applications

Future Generation Computer Systems
Deadline division-based heuristic for cost optimization in workflow scheduling

Information Sciences: an International Journal
Obtaining resource controllability in service cooperation environments

Proceedings of the 7th International Conference on Mobile and Ubiquitous Multimedia
On the Origin of Grid Species: The Living Application

ICCS '09 Proceedings of the 9th International Conference on Computational Science: Part I
Complex System Simulations with QosCosGrid

ICCS '09 Proceedings of the 9th International Conference on Computational Science: Part I
Developing Scientific Applications with Loosely-Coupled Sub-tasks

ICCS '09 Proceedings of the 9th International Conference on Computational Science: Part I
Improving Bounds on the Football Pool Problem by Integer Programming and High-Throughput Computing

INFORMS Journal on Computing
Scheduling Strategies for Cycle Scavenging in Multicluster Grid Systems

CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
On the Feasibility of Dynamically Scheduling DAG Applications on Shared Heterogeneous Systems

Euro-Par '09 Proceedings of the 15th International Euro-Par Conference on Parallel Processing
Energy aware scheduling on desktop grid environment with static performance prediction

SpringSim '09 Proceedings of the 2009 Spring Simulation Multiconference
A hayekian self-organization approach to service allocation in computing systems

Advanced Engineering Informatics
Utility driven dynamic resource allocation using competitive markets in computational grid

Advances in Engineering Software
Nephele: efficient parallel data processing in the cloud

Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers
VGrADS: enabling e-Science workflows on grids and clouds with fault tolerance

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
TeraGrid's integrated information service

Proceedings of the 5th Grid Computing Environments Workshop
Workflow management in the grid era: A goal-driven approach based on process patterns

Multiagent and Grid Systems - New tendencies on agents and grid environments
Predicting the Performance of a GRID Environment: An Initial Effort to Increase Scheduling Efficiency

FGIT '09 Proceedings of the 1st International Conference on Future Generation Information Technology
Secure resource control in service oriented applications

CCNC'09 Proceedings of the 6th IEEE Conference on Consumer Communications and Networking Conference
Grid load balancing using intelligent agents

Future Generation Computer Systems
JoiN: The implementation of a Java-based massively parallel grid

Future Generation Computer Systems - Special issue: Parallel computing technologies
PGGA: A predictable and grouped genetic algorithm for job scheduling

Future Generation Computer Systems - Parallel input/output management techniques (PIOMT) in cluster and grid computing
An adaptive task-level fault-tolerant approach to Grid

The Journal of Supercomputing
Dynamic heuristics for time and cost reduction in grid workflows

CSCWD'06 Proceedings of the 10th international conference on Computer supported cooperative work in design III
Experimental grid access for dynamic discovery and data transfer in distributed interactive simulation systems

ICCS'03 Proceedings of the 1st international conference on Computational science: PartI
Combining a high-throughput bioinformatics grid and bioinformatics web services

GCCB'06 Proceedings of the 2006 international conference on Distributed, high-performance and grid computing in computational biology
The Living Application: a Self-Organizing System for Complex Grid Tasks

International Journal of High Performance Computing Applications
A control-theoretic approach to automated local policy enforcement in computational grids

Future Generation Computer Systems
Automatic tuning in computational grids

PARA'06 Proceedings of the 8th international conference on Applied parallel computing: state of the art in scientific computing
Optimizing communications of data parallel programs in scalable cluster systems

GPC'08 Proceedings of the 3rd international conference on Advances in grid and pervasive computing
Middleware integration and deployment strategies for cyberinfrastructures

GPC'08 Proceedings of the 3rd international conference on Advances in grid and pervasive computing
Nonadiabatic Ab initio surface-hopping dynamics calculation in a grid environment - first experiences

ICCSA'07 Proceedings of the 2007 international conference on Computational science and its applications - Volume Part I
Multi-agent framework for performance-based resource management in computational grid environment

Multiagent and Grid Systems
Biased random walks on resource network graphs for load balancing

The Journal of Supercomputing
U-city portal for smart ubiquitous middleware

ICACT'10 Proceedings of the 12th international conference on Advanced communication technology
Experiences with resource provisioning for scientific workflows using Corral

Scientific Programming
Interoperability of BOINC and EGEE

Future Generation Computer Systems
End-To-End Cache System for Grid Computing: Design and Efficiency Analysis of a High-Throughput Bioinformatic Docking Application

International Journal of High Performance Computing Applications
Middleware support for many-task computing

Cluster Computing
SAGA BigJob: An Extensible and Interoperable Pilot-Job Abstraction for Distributed Applications and Systems

CCGRID '10 Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
PIVOT: An adaptive information discovery framework for computational grids

Information Sciences: an International Journal
Virtual Data System on distributed virtual machines in computational grids

International Journal of Ad Hoc and Ubiquitous Computing
A wide area network optimization service for effective access to heterogeneous datagrid infrastructures

Globe'10 Proceedings of the Third international conference on Data management in grid and peer-to-peer systems
Anticipated distributed task scheduling for grid environments

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
A fault avoidance strategy improving the reliability of the EGI production grid infrastructure

OPODIS'10 Proceedings of the 14th international conference on Principles of distributed systems
P-GRADE Portal: A generic workflow system to support user communities

Future Generation Computer Systems
Towards more effective utilization of computer systems

Proceedings of the 2nd ACM/SPEC International Conference on Performance engineering
Processing moldable tasks on the grid: Late job binding with lightweight user-level overlay

Future Generation Computer Systems
Practical applications in grid computing

SMO'05 Proceedings of the 5th WSEAS international conference on Simulation, modelling and optimization
A study on improving data transmission capability in grid using compression technology

SMO'05 Proceedings of the 5th WSEAS international conference on Simulation, modelling and optimization
Virtual workflow system for distributed collaborative scientific applications on Grids

Computers and Electrical Engineering
A hierarchical framework for cross-domain MapReduce execution

Proceedings of the second international workshop on Emerging computational methods for the life sciences
A multi-level scheduler for batch jobs on grids

The Journal of Supercomputing
Service monitoring and differentiation techniques for resource allocation in the grid, on the basis of the level of service

Future Generation Computer Systems
A CyberGIS gateway approach to interoperable access to the National Science Foundation TeraGrid and the Open Science Grid

Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery
Autonomic Resource Management with Support Vector Machines

GRID '11 Proceedings of the 2011 IEEE/ACM 12th International Conference on Grid Computing
Risk Management and Optimal Pricing in Online Storage Grids

Information Systems Research
Execution of a bioinformatics application in a joint IRISGrid/EGEE testbed

PPAM'05 Proceedings of the 6th international conference on Parallel Processing and Applied Mathematics
An effective information service architecture in grid environment

APPT'05 Proceedings of the 6th international conference on Advanced Parallel Processing Technologies
Integrating local job scheduler – LSFTM with GfarmTM

ISPA'05 Proceedings of the Third international conference on Parallel and Distributed Processing and Applications
Computational grid vs. parallel computer for coarse-grain parallelization of neural networks training

OTM'05 Proceedings of the 2005 OTM Confederated international conference on On the Move to Meaningful Internet Systems
Localization techniques for cluster-based data grid

ICA3PP'05 Proceedings of the 6th international conference on Algorithms and Architectures for Parallel Processing
Job scheduling policy for high throughput grid computing

ICA3PP'05 Proceedings of the 6th international conference on Algorithms and Architectures for Parallel Processing
Workflow overhead analysis and optimizations

Proceedings of the 6th workshop on Workflows in support of large-scale science
AME: an anyscale many-task computing engine

Proceedings of the 6th workshop on Workflows in support of large-scale science
Transparent migration of database services

SOFSEM'06 Proceedings of the 32nd conference on Current Trends in Theory and Practice of Computer Science
Grid application development on the basis of web portal technology

PARA'04 Proceedings of the 7th international conference on Applied Parallel Computing: state of the Art in Scientific Computing
M-Grid: using ubiquitous web technologies to create a computational grid

EGC'05 Proceedings of the 2005 European conference on Advances in Grid Computing
Workflow management in the crossgrid project

EGC'05 Proceedings of the 2005 European conference on Advances in Grid Computing
A multi-agent infrastructure and a service level agreement negotiation protocol for robust scheduling in grid computing

EGC'05 Proceedings of the 2005 European conference on Advances in Grid Computing
The gridkit distributed resource management framework

EGC'05 Proceedings of the 2005 European conference on Advances in Grid Computing
Localized communications of data parallel programs on multi-cluster grid systems

EGC'05 Proceedings of the 2005 European conference on Advances in Grid Computing
Applying the virtual data provenance model

IPAW'06 Proceedings of the 2006 international conference on Provenance and Annotation of Data
The cost of transparency: grid-based file access on the avaki data grid

ISPA'06 Proceedings of the 4th international conference on Parallel and Distributed Processing and Applications
Optimizations of data distribution localities in cluster grid environments

ICCSA'05 Proceedings of the 2005 international conference on Computational Science and Its Applications - Volume Part IV
A case for coordinated resource management in heterogeneous multicore platforms

ISCA'10 Proceedings of the 2010 international conference on Computer Architecture
Influences between performance based scheduling and service level agreements

Euro-Par'11 Proceedings of the 2011 international conference on Parallel Processing - Volume 2
Evolution of grid-based services for Diffusion Tensor Image analysis

Future Generation Computer Systems
Swift: A language for distributed parallel scripting

Parallel Computing
Towards a common model for pilot-jobs

Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing
Integration of Workflow Partitioning and Resource Provisioning

CCGRID '12 Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)
Efficient selection strategies towards processor reordering techniques for improving data locality in heterogeneous clusters

The Journal of Supercomputing
Composable reliability for asynchronous systems

USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference
Grid resource scheduling algorithm based on dynamic price-adjusting strategy

International Journal of Information and Communication Technology
Improving self-organized resource allocation with effective communication

AP2PC'08 Proceedings of the 7th international conference on Agents and Peer-to-Peer Computing
Cost-Aware and SLO-Fulfilling Software as a Service

Journal of Grid Computing
GJMF - a composable service-oriented grid job management framework

Future Generation Computer Systems
Evaluation of a utility computing model based on the federation of grid infrastructures

Euro-Par'07 Proceedings of the 13th international Euro-Par conference on Parallel Processing
Sigiri: uniform resource abstraction for grids and clouds

Concurrency and Computation: Practice & Experience
Model checking grid security

Future Generation Computer Systems
A semantic impact in decentralized resource discovery mechanism for grid computing environments

ICA3PP'12 Proceedings of the 12th international conference on Algorithms and Architectures for Parallel Processing - Volume Part II
Double auction-inspired meta-scheduling of parallel applications on global grids

Journal of Parallel and Distributed Computing
DAGwoman: enabling DAGMan-like workflows on non-Condor platforms

Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies
Efficiency analysis of decentralized grid scheduling with job migration and replication

Proceedings of the 7th International Conference on Ubiquitous Information Management and Communication
Effective hotspot removal system using neural network predictor

ACIIDS'13 Proceedings of the 5th Asian conference on Intelligent Information and Database Systems - Volume Part II
A Bee Colony based optimization approach for simultaneous job scheduling and data replication in grid environments

Computers and Operations Research
A survey of task mapping on production grids

ACM Computing Surveys (CSUR)
Using Gordon to accelerate LHC science

Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery
EventWave: programming model and runtime support for tightly-coupled elastic cloud applications

Proceedings of the 4th annual Symposium on Cloud Computing
Hopfield neural network for simultaneous job scheduling and data replication in grids

Future Generation Computer Systems
Green flexible opportunistic computing with task consolidation and virtualization

Cluster Computing
JETS: Language and System Support for Many-Parallel-Task Workflows

Journal of Grid Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

In recent years, there has been a dramatic increase in the number of available computing and storage resources. Yet few tools exist that allow these resources to be exploited effectively in an aggregated form. We present the Condor-G system, which leverages software from Globus and Condor to enable users to harness multi-domain resources as if they all belong to one personal domain. We describe the structure of Condor-G and how it handles job management, resource selection, security, and fault tolerance. We also present results from application experiments with the Condor-G system. We assert that Condor-G can serve as a general-purpose interface to Grid resources, for use by both end users and higher-level program development tools.