Integrating a stream processing engine and databases for persistent streaming data management

  • Authors:

  • Yousuke Watanabe;Shinichi Yamada;Hiroyuki Kitagawa;Toshiyuki Amagasa

  • Affiliations:

  • Japan Science and Technology Agency;Graduate School of Systems and Information Engineering, University of Tsukuba;Graduate School of Systems and Information Engineering, University of Tsukuba and Center of Computational Sciences, University of Tsukuba;Graduate School of Systems and Information Engineering, University of Tsukuba and Center of Computational Sciences, University of Tsukuba

  • Venue:
  • DEXA'07 Proceedings of the 18th international conference on Database and Expert Systems Applications
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

Because of increased stream data, managing stream data has become quite important. This paper describes our data stream management system, which employs an architecture combining a stream processing engine and DBMS. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Our proposed query language supports not only filtering, join, and projection over data streams, but also continuous persistence requirements for stream data. Users can also specify continuous queries that integrate streaming data and historical data stored in DBMS. Another contribution of this paper is feasibility validation of queries. Processing queries on streams with frequent inputs may cause the system to overflow its capacity. Specifically, the maximum writing rate to DBMS is a significant bottleneck when we try to store stream data into DBMS. Our system detects infeasible queries in advance.