Local correlation detection with linearity enhancement in streaming data

  • Authors:
  • Qing Xie;Shuo Shang;Bo Yuan;Chaoyi Pang;Xiangliang Zhang

  • Affiliations:
  • King Abdullah University of Science and Technology, Thuwal, Saudi Arabia;China University of Petroleum-Beijing, Beijing, China;Tsinghua University, Shenzhen, China;The Commonwealth Scientific and Industrial Research Organisation, Brisbane, Australia;King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

  • Venue:
  • Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper addresses the challenges in detecting the potential correlation between numerical data streams, which facilitates the research of data stream mining and pattern discovery. We focus on local correlation with delay, which may occur in burst at different time in different streams, and last for a limited period. The uncertainty on the correlation occurrence and the time delay make it difficult to monitor the correlation online. Furthermore, the conventional correlation measure lacks the ability of reflecting visual linearity, which is more desirable in reality. This paper proposes effective methods to continuously detect the correlation between data streams. Our approach is based on the Discrete Fourier Transform to make rapid cross-correlation calculation with time delay allowed. In addition, we introduce a shape-based similarity measure into the framework, which refines the results by representative trend patterns to enhance the significance of linearity. The similarity of proposed linear representations can quickly estimate the correlation, and the window sliding strategy in segment level improves the efficiency for online detection. The empirical study demonstrates the accuracy of our detection approach, as well as more than $30\%$ improvement of efficiency.