The extensibility framework in Microsoft StreamInsight

Authors:
Mohamed Ali;Badrish Chandramouli;Jonathan Goldstein;Roman Schindlauer
Affiliations:
Microsoft Corporation, One Microsoft Way, Redmond WA 98052, USA;Microsoft Research, One Microsoft Way, Redmond WA 98052, USA;Microsoft Corporation, One Microsoft Way, Redmond WA 98052, USA;Microsoft Corporation, One Microsoft Way, Redmond WA 98052, USA
Venue:
ICDE '11 Proceedings of the 2011 IEEE 27th International Conference on Data Engineering
Year:
2011

Citing 0
Cited 4

An extensibility approach for spatio-temporal stream processing using microsoft streaminsight

SSTD'11 Proceedings of the 12th international conference on Advances in spatial and temporal databases
OCEANUS: a spatio-temporal data stream system prototype

Proceedings of the Third ACM SIGSPATIAL International Workshop on GeoStreaming
Safe step: a real-time gps tracking and analysis system for criminal activities using ankle bracelets

Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems
Towards window stream queries over continuous phenomena

Proceedings of the 4th ACM SIGSPATIAL International Workshop on GeoStreaming

Quantified Score

Hi-index	0.00

Visualization

Abstract

Microsoft StreamInsight (StreamInsight, for brevity) is a platform for developing and deploying streaming applications, which need to run continuous queries over high-data-rate streams of input events. StreamInsight leverages a well-defined temporal stream model and operator algebra, as the underlying basis for processing long-running continuous queries over event streams. This allows StreamInsight to handle imperfections in event delivery and to provide correctness guarantees on the generated output. StreamInsight natively supports a diverse range of off-the-shelf streaming operators. In order to cater to a much broader range of customer scenarios and applications, StreamInsight has recently introduced a new extensibility infrastructure. With this infrastructure, StreamInsight enables developers to integrate their domain expertise within the query pipeline in the form of user defined modules (functions, operators, and aggregates). This paper describes the extensibility framework in StreamInsight; an ongoing effort at Microsoft SQL Server to support the integration of user-defined modules in a stream processing system. More specifically, the paper addresses the extensibility problem from three perspectives: the query writer's perspective, the user defined module writer's perspective, and the system's internal perspective. The paper introduces and addresses a range of new and subtle challenges that arise when we try to add extensibility to a streaming system, in a manner that is easy to use, powerful, and practical. We summarize our experience and provide future directions for supporting stream-oriented workloads in different business domains.