High throughput computing over peer-to-peer networks
Future Generation Computer Systems
| Hi-index | 0.00 |
Scheduling large amounts of tasks in distributed computing platforms composed of millions of nodes is a challenging goal, even more in a fully decentralized way and with low overhead. Thus, we propose a new scalable scheduler for task workflows with deadlines following a completely decentralized architecture. It's built upon a tree-based P2P overlay that supports efficient and fast aggregation of resource availability information. Constraints for deadlines and the correct timing of tasks in workflows are guaranteed with a suitable distributed management of availability time intervals of resources. A local scheduler in each node provides its available time intervals to the distributed global scheduler, which summarizes them in the aggregation process. A two phase reservation protocol looks for suitable resources that comply with workflow structure and deadline. Experimental results, from simulations of a system composed of one million nodes, show scalable fast scheduling with low overhead that can allow a high dynamic usage of computational resources.