On the execution of parallel programs on multiprocessor systems—a queuing theory approach
Journal of the ACM (JACM)
Optimization flow control—I: basic algorithm and convergence
IEEE/ACM Transactions on Networking (TON)
Fair end-to-end window-based congestion control
IEEE/ACM Transactions on Networking (TON)
GridFlow: Workflow Management for Grid Computing
CCGRID '03 Proceedings of the 3st International Symposium on Cluster Computing and the Grid
End-to-end congestion control schemes: utility functions, random losses and ECN marks
IEEE/ACM Transactions on Networking (TON)
Maximum Pressure Policies in Stochastic Processing Networks
Operations Research
Maximizing Queueing Network Utility Subject to Stability: Greedy Primal-Dual Algorithm
Queueing Systems: Theory and Applications
Adaptive Control of Extreme-scale Stream Processing Systems
ICDCS '06 Proceedings of the 26th IEEE International Conference on Distributed Computing Systems
Load shedding and distributed resource control of stream processing networks
Performance Evaluation
Load shedding in a data stream manager
VLDB '03 Proceedings of the 29th international conference on Very large data bases - Volume 29
Stable and utility-maximizing scheduling for stochastic processing networks
Allerton'09 Proceedings of the 47th annual Allerton conference on Communication, control, and computing
Distributed resource allocation for synchronous fork and join processing networks
INFOCOM'10 Proceedings of the 29th conference on Information communications
Joint congestion control, routing, and MAC for stability and fairness in wireless networks
IEEE Journal on Selected Areas in Communications
Linear-speed interior-path algorithms for distributed control of information networks
Performance Evaluation
Utility optimal scheduling in processing networks
Performance Evaluation
Towards a queueing-based framework for in-network function computation
Queueing Systems: Theory and Applications
On distributed computation rate optimization for deploying cloud computing programming frameworks
ACM SIGMETRICS Performance Evaluation Review
Joint optimization of overlapping phases in MapReduce
Performance Evaluation
| Hi-index | 0.00 |
This paper addresses the problem of distributed resource allocation in general fork and join processing networks. The problem is motivated by the complicated processing requirements arising from distributed data intensive computing. In such applications, the underlying data processing software consists of a rich set of semantics that include synchronous and asynchronous data fork and data join. The different types of semantics and processing requirements introduce complex interdependence between various data flows within the network. We study the distributed resource allocation problem in such systems with the goal of achieving the maximum total utility of output streams. Past research has dealt with networks with specific types of fork/join semantics, but none of them included all four types. We propose a novel modeling framework that can represent all combinations of fork and join semantics, and formulate the resource allocation problem as a convex optimization problem on this model. We propose a shadow-queue based decentralized iterative algorithm to solve the resource allocation problem. We show that the algorithm guarantees optimality and demonstrate through simulation that it can adapt quickly to dynamically changing environments.