Multicommodity max-flow min-cut theorems and their use in designing approximation algorithms
Journal of the ACM (JACM)
Computers and Intractability: A Guide to the Theory of NP-Completeness
Computers and Intractability: A Guide to the Theory of NP-Completeness
StreamIt: A Language for Streaming Applications
CC '02 Proceedings of the 11th International Conference on Compiler Construction
The VLDB Journal — The International Journal on Very Large Data Bases
Position: short object lifetimes require a delete-optimized storage system
Proceedings of the 11th workshop on ACM SIGOPS European workshop
Design, implementation, and evaluation of the linear road bnchmark on the stream processing core
Proceedings of the 2006 ACM SIGMOD international conference on Management of data
Towards Autonomic Fault Recovery in System-S
ICAC '07 Proceedings of the Fourth International Conference on Autonomic Computing
SPC: a distributed, scalable platform for data mining
Proceedings of the 4th international workshop on Data mining standards, services and platforms
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Storage optimization for large-scale distributed stream-processing systems
ACM Transactions on Storage (TOS)
Faster and Simpler Algorithms for Multicommodity Flow and Other Fractional Packing Problems
SIAM Journal on Computing
SPADE: the system s declarative stream processing engine
Proceedings of the 2008 ACM SIGMOD international conference on Management of data
SODA: an optimizing scheduler for large-scale stream-based distributed computer systems
Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware
XStream: a Signal-Oriented Data Stream Management System
ICDE '08 Proceedings of the 2008 IEEE 24th International Conference on Data Engineering
Scale-Up Strategies for Processing High-Rate Data Streams in System S
ICDE '09 Proceedings of the 2009 IEEE International Conference on Data Engineering
A code generation approach to optimizing high-performance distributed data stream processing
Proceedings of the 18th ACM conference on Information and knowledge management
Job Admission and Resource Allocation in Distributed Streaming Systems
Job Scheduling Strategies for Parallel Processing
On the NP-Completeness of some graph cluster measures
SOFSEM'06 Proceedings of the 32nd conference on Current Trends in Theory and Practice of Computer Science
IBM infosphere streams for scalable, real-time, intelligent transportation services
Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
Design principles for developing stream processing applications
Software—Practice & Experience - Focus on Selected PhD Literature Reviews in the Practical Aspects of Software Technology
Visual debugging for stream processing applications
RV'10 Proceedings of the First international conference on Runtime verification
Hirundo: a mechanism for automated production of optimized data stream graphs
ICPE '12 Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering
From a calculus to an execution environment for stream processing
Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems
Building user-defined runtime adaptation routines for stream processing applications
Proceedings of the VLDB Endowment
Tutorial: stream processing optimizations
Proceedings of the 7th ACM international conference on Distributed event-based systems
A catalog of stream processing optimizations
ACM Computing Surveys (CSUR)
Automatic optimization of stream programs via source program operator graph transformations
Distributed and Parallel Databases
IBM streams processing language: analyzing big data in motion
IBM Journal of Research and Development
Hi-index | 0.00 |
In this paper, we describe an optimization scheme for fusing compile-time operators into reasonably-sized run-time software units called processing elements (PEs). Such PEs are the basic deployable units in System S, a highly scalable distributed stream processing middleware system. Finding a high quality fusion significantly benefits the performance of streaming jobs. In order to maximize throughput, our solution approach attempts to minimize the processing cost associated with inter-PE stream traffic while simultaneously balancing load across the processing hosts. Our algorithm computes a hierarchical partitioning of the operator graph based on a minimum-ratio cut subroutine. We also incorporate several fusion constraints in order to support real-world SYSTEM S jobs. We experimentally compare our algorithm with several other reasonable alternative schemes, highlighting the effectiveness of our approach.