Models and issues in data stream systems
Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Gigascope: high performance network monitoring with an SQL interface
Proceedings of the 2002 ACM SIGMOD international conference on Management of data
Data streams: algorithms and applications
SODA '03 Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms
Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Total
ICDE '96 Proceedings of the Twelfth International Conference on Data Engineering
Exploiting Punctuation Semantics in Continuous Data Streams
IEEE Transactions on Knowledge and Data Engineering
Gigascope: a stream database for network applications
Proceedings of the 2003 ACM SIGMOD international conference on Management of data
Fjording the Stream: An Architecture for Queries Over Streaming Sensor Data
ICDE '02 Proceedings of the 18th International Conference on Data Engineering
Flexible time management in data stream systems
PODS '04 Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems
Sampling algorithms in a stream operator
Proceedings of the 2005 ACM SIGMOD international conference on Management of data
Semantics and evaluation techniques for window aggregates in data streams
Proceedings of the 2005 ACM SIGMOD international conference on Management of data
Tribeca: a system for managing large databases of network traffic
ATEC '98 Proceedings of the annual conference on USENIX Annual Technical Conference
Monitoring streams: a new class of data management applications
VLDB '02 Proceedings of the 28th international conference on Very Large Data Bases
Data stream query processing: a tutorial
VLDB '03 Proceedings of the 29th international conference on Very large data bases - Volume 29
Minimizing latency and memory in DSMS: a unified approach to quasi-optimal scheduling
SSPS '08 Proceedings of the 2nd international workshop on Scalable stream processing system
Wide-scale data stream management
ATC'08 USENIX 2008 Annual Technical Conference on Annual Technical Conference
Out-of-order processing: a new architecture for high-performance stream systems
Proceedings of the VLDB Endowment
Anomaly-free incremental output in stream processing
Proceedings of the 17th ACM conference on Information and knowledge management
Data-driven memory management for stream join
Information Systems
Window Update Patterns in Stream Operators
ADBIS '09 Proceedings of the 13th East European Conference on Advances in Databases and Information Systems
Distributed event stream processing with non-deterministic finite automata
Proceedings of the Third ACM International Conference on Distributed Event-Based Systems
On-the-fly progress detection in iterative stream queries
Proceedings of the VLDB Endowment
Toward Formal Semantics for Data and Schema Evolution in Data Stream Management Systems
ER '09 Proceedings of the ER 2009 Workshops (CoMoL, ETheCoM, FP-UML, MOST-ONISW, QoIS, RIGiM, SeCoGIS) on Advances in Conceptual Modeling - Challenging Perspectives
Proceedings of the international conference on Multimedia information retrieval
An extensible test framework for the Microsoft StreamInsight query processor
Proceedings of the Third International Workshop on Testing Database Systems
Relational languages and data models for continuous queries on sequences and data streams
ACM Transactions on Database Systems (TODS)
Bistro data feed management system
Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
Streams, security and scalability
DBSec'05 Proceedings of the 19th annual IFIP WG 11.3 working conference on Data and Applications Security
Window specification over data streams
EDBT'06 Proceedings of the 2006 international conference on Current Trends in Database Technology
Logical foundations of continuous query languages for data streams
Datalog 2.0'12 Proceedings of the Second international conference on Datalog in Academia and Industry
Proceedings of the 13th International Middleware Conference
MillWheel: fault-tolerant stream processing at internet scale
Proceedings of the VLDB Endowment
| Hi-index | 0.00 |
Data stream management systems often rely on ordering properties of tuple attributes in order to implement non-blocking operators. However, query operators that work with multiple streams, such as stream merge or join, can often still block if one of the input stream is very slow or bursty. In principle, punctuation and heartbeat mechanisms have been proposed to unblock streaming operators. In practice, it is a challenge to incorporate such mechanisms into a high-performance stream management system that is operational in an industrial application.In this paper, we introduce a system for punctuation-carrying heartbeat generation that we developed for Gigascope, a high-performance streaming database for network monitoring, that is operationally used within AT&T's IP backbone. We show how heartbeats can be regularly generated by low-level nodes in query execution plans and propagated upward unblocking all streaming operators on its way. Additionally, our heartbeat mechanism can be used for other applications in distributed settings such as detecting node failures, performance monitoring, and query optimization. A performance evaluation using live data feeds shows that our system is capable of working at multiple Gigabit line speeds in a live, industrial deployment and can significantly decrease the query memory utilization.