Integer and combinatorial optimization
Integer and combinatorial optimization
Resource allocation problems: algorithmic approaches
Resource allocation problems: algorithmic approaches
Introduction to algorithms
Scheduling in Computer and Manufacturing Systems
Scheduling in Computer and Manufacturing Systems
Dynamic Load Distribution in the Borealis Stream Processor
ICDE '05 Proceedings of the 21st International Conference on Data Engineering
Network-Aware Operator Placement for Stream-Processing Systems
ICDE '06 Proceedings of the 22nd International Conference on Data Engineering
Position: short object lifetimes require a delete-optimized storage system
Proceedings of the 11th workshop on ACM SIGOPS European workshop
Design, implementation, and evaluation of the linear road bnchmark on the stream processing core
Proceedings of the 2006 ACM SIGMOD international conference on Management of data
Providing resiliency to load variations in distributed stream processing
VLDB '06 Proceedings of the 32nd international conference on Very large data bases
Towards Autonomic Fault Recovery in System-S
ICAC '07 Proceedings of the Fourth International Conference on Autonomic Computing
Distributed Resource Management and Admission Control of Stream Processing Systems with Max Utility
ICDCS '07 Proceedings of the 27th International Conference on Distributed Computing Systems
SPC: a distributed, scalable platform for data mining
Proceedings of the 4th international workshop on Data mining standards, services and platforms
Staying FIT: efficient load shedding techniques for distributed stream processing
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Storage optimization for large-scale distributed stream-processing systems
ACM Transactions on Storage (TOS)
Biologically-inspired distributed middleware management for stream processing systems
Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware
Biologically-inspired distributed middleware management for stream processing systems
Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware
SLIPstream: scalable low-latency interactive perception on streaming data
Proceedings of the 18th international workshop on Network and operating systems support for digital audio and video
Proceedings of the 2nd Workshop on High Performance Computational Finance
Proceedings of the 18th ACM conference on Information and knowledge management
COLA: optimizing stream processing applications via graph partitioning
Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware
An empirical study of high availability in stream processing systems
Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware
MARIO: middleware for assembly and deployment of multi-platform flow-based applications
Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware
Exploiting multi-level parallelism for low-latency activity recognition in streaming video
MMSys '10 Proceedings of the first annual ACM SIGMM conference on Multimedia systems
Visualizing large-scale streaming applications
Information Visualization
IBM infosphere streams for scalable, real-time, intelligent transportation services
Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
VLAN-based routing infrastructure for an all-optical circuit switched LAN
GLOBECOM'09 Proceedings of the 28th IEEE conference on Global telecommunications
COLA: optimizing stream processing applications via graph partitioning
Middleware'09 Proceedings of the ACM/IFIP/USENIX 10th international conference on Middleware
Scalable performance of system S for extract-transform-load processing
Proceedings of the 3rd Annual Haifa Experimental Systems Conference
Placement of replicated tasks for distributed stream processing systems
Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems
Pattern rewriting framework for event processing optimization
Proceedings of the 5th ACM international conference on Distributed event-based system
Distributed middleware reliability and fault tolerance support in system S
Proceedings of the 5th ACM international conference on Distributed event-based system
Processing flows of information: From data stream to complex event processing
ACM Computing Surveys (CSUR)
Highly scalable speech processing on data stream management system
DASFAA'12 Proceedings of the 17th international conference on Database Systems for Advanced Applications - Volume Part II
CCGRID '12 Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)
From a calculus to an execution environment for stream processing
Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems
Tutorial: stream processing optimizations
Proceedings of the 7th ACM international conference on Distributed event-based systems
A catalog of stream processing optimizations
ACM Computing Surveys (CSUR)
A performance analysis of system s, s4, and esper via two level benchmarking
QEST'13 Proceedings of the 10th international conference on Quantitative Evaluation of Systems
Supporting distributed feed-following apps over edge devices
Proceedings of the VLDB Endowment
Active workflow system for near real-time extreme-scale science
Proceedings of the first workshop on Parallel programming for analytics applications
An evaluation of zookeeper for high availability in system S
Proceedings of the 5th ACM/SPEC international conference on Performance engineering
| Hi-index | 0.01 |
This paper describes the SODA scheduler for System S, a highly scalable distributed stream processing system. Unlike traditional batch applications, streaming applications are open-ended. The system cannot typically delay the processing of the data. The scheduler must be able to shift resource allocation dynamically in response to changes to resource availability, job arrivals and departures, incoming data rates and so on. The design assumptions of System S, in particular, pose additional scheduling challenges. SODA must deal with a highly complex optimization problem, which must be solved in real-time while maintaining scalability. SODA relies on a careful problem decomposition, and intelligent use of both heuristic and exact algorithms. We describe the design and functionality of SODA, outline the mathematical components, and describe experiments to show the performance of the scheduler.