Balancing reducer workload for skewed data using sampling-based partitioning
Computers and Electrical Engineering
| Hi-index | 0.00 |
Since the generation of detailed traffic statistics does not scale well with link speed, increasingly passive traffic measurement employs sampling at the packet or flow level. Sampling has become an attractive and scalable means to measure flow data on high-speed links. However, knowing the length distributions of traffic flows passing through a network link is useful for some applications such as inferring traffic demands, characterizing source traffic, and detecting traffic anomalies. Passive traffic measurement increasingly makes inferences from sampled network traffic. However, previous work has shown the inaccuracy of estimating flow length distributions from sampled traffic when the sampling is performed at the packet level. In this paper, we propose a novel method that uses flow statistics formed from double-sampled packet stream to infer the absolute frequencies of lengths of flows in the unsampled stream. We achieve this through statistical inference and by exploiting heavy-tailed feather. The method allow us to recover the complete flow length distribution.