MapReduce: simplified data processing on large clusters
OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Proportional-share scheduling for distributed storage systems
FAST '07 Proceedings of the 5th USENIX conference on File and Storage Technologies
The Hadoop Distributed File System
MSST '10 Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST)
| Hi-index | 0.00 |
Existing big-data systems (e.g., Hadoop/MapReduce) do not expose management of shared storage I/O resources. As such, application's performance may degrade in unpredictable ways under I/O contention, even with fair sharing of computing resources. This paper proposes \emph{IBIS}, a new Interposed Big-data I/O Scheduler, to provide performance differentiation for competing applications' I/Os in a shared MapReduce-type big-data system. IBIS is implemented in Hadoop by interposing HDFS I/Os and use an SFQ-based proportional-sharing algorithm. Experiments show that the IBIS provides strong performance isolation for one application against another highly I/O-intensive application. IBIS also enforces good proportional sharing of the global bandwidth among competing parallel applications, by coordinating distributed IBIS schedulers to deal with the uneven distribution of local services in big-data systems.