Benchmarking Anomaly-Based Detection Systems

  • Authors:
  • Roy A. Maxion;Kymie M. C. Tan

  • Affiliations:
  • -;-

  • Venue:
  • DSN '00 Proceedings of the 2000 International Conference on Dependable Systems and Networks (formerly FTCS-30 and DCCA-8)
  • Year:
  • 2000

Quantified Score

Hi-index 0.00

Visualization

Abstract

Anomaly detection is a key element of intrusion-detection and other detection systems in which perturbations of normal behavior suggest the presence of intentionally or unintentionally induced attacks, faults, defects, etc. Because most anomaly detectors are based on probabilistic algorithms that exploit the intrinsic structure, or regularity, embedded in data logs, a fundamental question is whether such structure influences detection performance. If detector performance were indeed a function of environmental regularity, it would be critical to match detectors to environmental characteristics. In intrusion-detection settings, however, this is not done, possibly because such characteristics are not easily ascertained. This paper introduces a metric for characterizing structure in data environments, and tests the hypothesis that intrinsic structure influences probabilistic detection. In a series of experiments, an anomaly-detection algorithm was applied to a benchmark suite of 165 carefully calibrated, anomaly-injected datasets of varying structure. Results showed performance differences of as much as an order of magnitude, indicating that current approaches to anomaly detection may not be universally dependable.