SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
MapReduce: simplified data processing on large clusters
OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
| Hi-index | 0.00 |
Batch jobs are the back bone of IT functionality in all industries. Typically these jobs are executed overnight, after "End of Day/Week/Month/Year" and perform business critical processing of huge amounts of data, to bring software systems up-to-date for the next day's business. Due to the huge processing requirements, batch systems have been traditionally implemented in legacy Mainframe environments. These systems are costly to develop and maintain, and require manpower skilled in what is fast becoming an obsolete platform. In this paper, I propose HiPERS Batch- a simple multi-server batch framework which: (i) is High in Performance -- can match the processing power of legacy batch systems, (ii) is Extensible - accommodates new batch business functionalities without code change, (iii) is Reliable -- recovers from failures and resumes processing from the correct point, and (iv) is Scalable -- easily handles growing volumes of business data by accommodating increasing number of servers in its configuration without performance degradation. HiPERS Batch exploits the data and task level parallelism inherent in batch functionality. A salient feature of HiPERS multi server batch framework is that the communication between the servers takes place solely through the database -- this does not impact performance as is commonly believed. As part of work done for a large Insurer, HiPERS Batch was implemented using Microsoft"s.NET framework 2.0 (HiPERS Batch.NET) with virtual servers of modest specifications (no more powerful than a modern day laptop) to replace a powerful legacy mainframe batch system; the results of this implementation, as shown in this paper, prove that HiPERS Batch is a viable framework for a modern batch system.