Detecting distributed scans using high-performance query-driven visualization

  • Authors:

  • Kurt Stockinger;E. Wes Bethel;Scott Campbell;Eli Dart;Kesheng Wu

  • Affiliations:

  • Lawrence Berkeley National Laboratory, University of California;Lawrence Berkeley National Laboratory, University of California;Lawrence Berkeley National Laboratory, University of California;Lawrence Berkeley National Laboratory, University of California;Lawrence Berkeley National Laboratory, University of California

  • Venue:
  • Proceedings of the 2006 ACM/IEEE conference on Supercomputing
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

Modern forensic analytics applications, like network traffic analysis, perform high-performance hypothesis testing, knowledge discovery and data mining on very large datasets. One essential strategy to reduce the time required for these operations is to select only the most relevant data records for a given computation. In this paper, we present a set of parallel algorithms that demonstrate how an efficient selection mechanism -- bitmap indexing -- significantly speeds up a common analysis task, namely, computing conditional histogram on very large datasets. We present a thorough study of the performance characteristics of the parallel conditional histogram algorithms. As a case study, we compute conditional histograms for detecting distributed scans hidden in a dataset consisting of approximately 2.5 billion network connection records. We show that these conditional histograms can be computed on interactive time scale (i.e., in seconds). We also show how to progressively modify the selection criteria to narrow the analysis and find the sources of the distributed scans.