Adaptive parallel aggregation algorithms
SIGMOD '95 Proceedings of the 1995 ACM SIGMOD international conference on Management of data
On the analysis of randomized load balancing schemes
Proceedings of the ninth annual ACM symposium on Parallel algorithms and architectures
Cluster-based scalable network services
Proceedings of the sixteenth ACM symposium on Operating systems principles
Locality-aware request distribution in cluster-based network servers
Proceedings of the eighth international conference on Architectural support for programming languages and operating systems
MagPIe: MPI's collective communication operations for clustered wide area systems
Proceedings of the seventh ACM SIGPLAN symposium on Principles and practice of parallel programming
Proceedings of the seventeenth ACM symposium on Operating systems principles
Optimizing threaded MPI execution on SMP clusters
ICS '01 Proceedings of the 15th international conference on Supercomputing
SEDA: an architecture for well-conditioned, scalable internet services
SOSP '01 Proceedings of the eighteenth ACM symposium on Operating systems principles
MPI: The Complete Reference
Efficient Collective Communication on Heterogeneous Networks of Workstations
ICPP '98 Proceedings of the 1998 International Conference on Parallel Processing
SWEB: Towards a Scalable World Wide Web Server on Multicomputers
IPPS '96 Proceedings of the 10th International Parallel Processing Symposium
Cluster Load Balancing for Fine-Grain Network Services
IPDPS '02 Proceedings of the 16th International Parallel and Distributed Processing Symposium
Ninja: A Framework for Network Services
ATEC '02 Proceedings of the General Track of the annual conference on USENIX Annual Technical Conference
Harvest, Yield, and Scalable Tolerant Systems
HOTOS '99 Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems
Exploiting Hierarchy in Parallel Computer Networks to Optimize Collective Operation Performance
IPDPS '00 Proceedings of the 14th International Symposium on Parallel and Distributed Processing
TAG: a Tiny AGgregation service for Ad-Hoc sensor networks
OSDI '02 Proceedings of the 5th symposium on Operating systems design and implementationCopyright restrictions prevent ACM from being able to make the PDFs for this conference available for downloading
Integrated resource management for cluster-based internet services
OSDI '02 Proceedings of the 5th symposium on Operating systems design and implementationCopyright restrictions prevent ACM from being able to make the PDFs for this conference available for downloading
Neptune: scalable replication management and programming support for cluster-based network services
USITS'01 Proceedings of the 3rd conference on USENIX Symposium on Internet Technologies and Systems - Volume 3
The multispace: an evolutionary platform for infrastructural services
ATEC '99 Proceedings of the annual conference on USENIX Annual Technical Conference
Flash: an efficient and portable web server
ATEC '99 Proceedings of the annual conference on USENIX Annual Technical Conference
SCOPE: easy and efficient parallel processing of massive data sets
Proceedings of the VLDB Endowment
Integrating hadoop and parallel DBMs
Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
OSDI'08 Proceedings of the 8th USENIX conference on Operating systems design and implementation
Proceedings of the second international workshop on MapReduce and its applications
Network-Based inference algorithm on hadoop
ISMIS'12 Proceedings of the 20th international conference on Foundations of Intelligent Systems
Hi-index | 0.00 |
Large-scale cluster-based Internet services often host partitioned datasets to provide incremental scalability. The aggregation of results produced from multiple partitions is a fundamental building block for the delivery of these services. This paper presents the design and implementation of a programming primitive -- Data Aggregation Call (DAC) -- to exploit partition parallelism for cluster-based Internet services. A DAC request specifies a local processing operator and a global reduction operator, and it aggregates the local processing results from participating nodes through the global reduction operator. Applications may allow a DAC request to return partial aggregation results as a tradeoff between quality and availability. Our architecture design aims at improving interactive responses with sustained throughput for typical cluster environments where platform heterogeneity and software/hardware failures are common. At the cluster level, our load-adaptive reduction tree construction algorithm balances processing and aggregation load across servers while exploiting partition parallelism. Inside each node, we employ an event-driven thread pool design that prevents slow nodes from adversely affecting system throughput under highly concurrent workload. We further devise a staged timeout scheme that eagerly prunes slow or unresponsive servers from the reduction tree to meet soft deadlines. We have used the DAC primitive to implement several applications: a search engine document retriever, a parallel protein sequence matcher, and an online parallel facial recognizer. Our experimental and simulation results validate the effectiveness of the proposed optimization techniques for reducing response time, improving throughput, and gracefully handling server unresponsiveness. We also demonstrate the ease-of use of the DAC primitive and the scalability of our architecture design.