Uncovering Global Icebergs in Distributed Streams: Results and Implications

  • Authors:

  • Guanyao Huang;Ashwin Lall;Chen-Nee Chuah;Jun Xu

  • Affiliations:

  • Department of Electrical and Computer Engineering, University of California Davis, Davis, USA 95616;Georgia Institute of Technology, College of Computing, Atlanta, USA 30332-0280;Department of Electrical and Computer Engineering, University of California Davis, Davis, USA 95616;Georgia Institute of Technology, College of Computing, Atlanta, USA 30332-0280

  • Venue:
  • Journal of Network and Systems Management
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Discovering icebergs in distributed streams of data is an important problem for a number of applications in networking and databases. While previous work has concentrated on measuring these icebergs in the non-distributed streaming case or in the non-streaming distributed case, we present a general framework that allows for distributed processing across multiple streams of data. We compare several of the state-of-the-art streaming algorithms for estimating local elephants in the individual streams. However, since an iceberg may be hidden by being distributed across many different streams, we add a sampling component to handle such cases. We provide a novel taxonomy of current sketches and perform a thorough analysis of the strengths and weaknesses of each scheme under various QoS metrics, using both real and synthetic Internet trace data. We summarize their performance and discuss the implications for the future design of sketches.