Multicommodity max-flow min-cut theorems and their use in designing approximation algorithms
Journal of the ACM (JACM)
StreamIt: A Language for Streaming Applications
CC '02 Proceedings of the 11th International Conference on Compiler Construction
TelegraphCQ: continuous dataflow processing
Proceedings of the 2003 ACM SIGMOD international conference on Management of data
The VLDB Journal — The International Journal on Very Large Data Bases
Position: short object lifetimes require a delete-optimized storage system
Proceedings of the 11th workshop on ACM SIGOPS European workshop
Design, implementation, and evaluation of the linear road bnchmark on the stream processing core
Proceedings of the 2006 ACM SIGMOD international conference on Management of data
Towards Autonomic Fault Recovery in System-S
ICAC '07 Proceedings of the Fourth International Conference on Autonomic Computing
SPC: a distributed, scalable platform for data mining
Proceedings of the 4th international workshop on Data mining standards, services and platforms
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Storage optimization for large-scale distributed stream-processing systems
ACM Transactions on Storage (TOS)
Faster and Simpler Algorithms for Multicommodity Flow and Other Fractional Packing Problems
SIAM Journal on Computing
SPADE: the system s declarative stream processing engine
Proceedings of the 2008 ACM SIGMOD international conference on Management of data
Scheduling: Theory, Algorithms, and Systems
Scheduling: Theory, Algorithms, and Systems
SODA: an optimizing scheduler for large-scale stream-based distributed computer systems
Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware
XStream: a Signal-Oriented Data Stream Management System
ICDE '08 Proceedings of the 2008 IEEE 24th International Conference on Data Engineering
Scale-Up Strategies for Processing High-Rate Data Streams in System S
ICDE '09 Proceedings of the 2009 IEEE International Conference on Data Engineering
A code generation approach to optimizing high-performance distributed data stream processing
Proceedings of the 18th ACM conference on Information and knowledge management
Job Admission and Resource Allocation in Distributed Streaming Systems
Job Scheduling Strategies for Parallel Processing
On the NP-Completeness of some graph cluster measures
SOFSEM'06 Proceedings of the 32nd conference on Current Trends in Theory and Practice of Computer Science
Pattern rewriting framework for event processing optimization
Proceedings of the 5th ACM international conference on Distributed event-based system
Processing flows of information: From data stream to complex event processing
ACM Computing Surveys (CSUR)
Understanding and improving the cost of scaling distributed event processing
Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems
Software—Practice & Experience
Auto-parallelizing stateful distributed streaming applications
Proceedings of the 21st international conference on Parallel architectures and compilation techniques
Active workflow system for near real-time extreme-scale science
Proceedings of the first workshop on Parallel programming for analytics applications
| Hi-index | 0.00 |
In this paper, we describe an optimization scheme for fusing compile-time operators into reasonably-sized run-time software units called processing elements (PEs). Such PEs are the basic deployable units in System S, a highly scalable distributed stream processing middleware system. Finding a high quality fusion significantly benefits the performance of streaming jobs. In order to maximize throughput, our solution approach attempts to minimize the processing cost associated with inter-PE stream traffic while simultaneously balancing load across the processing hosts. Our algorithm computes a hierarchical partitioning of the operator graph based on a minimum-ratio cut subroutine. We also incorporate several fusion constraints in order to support real-world System S jobs. We experimentally compare our algorithm with several other reasonable alternative schemes, highlighting the effectiveness of our approach.