Provenance for MapReduce-based data-intensive workflows
Proceedings of the 6th workshop on Workflows in support of large-scale science
| Hi-index | 0.00 |
To execute workflows on a compute cluster resource, workflow engines can work with cluster resource manager software to distribute jobs into compute nodes on the cluster. We discuss how to interact with traditional Oracle Grid Engine and recent Hadoop cluster resource managers using a dataflow-based scheduling approach to balance compute resource load for data-parallel workflow execution. Our experiments show that: 1) The presented approach can balance computational resource load well by interacting with the resource managers and provides good execution performance on both physical and virtual clusters, 2) Oracle Grid Engine outperforms Hadoop for CPU-intensive applications on small-scale clusters.