Self-adaptive and dynamic clustering for online anomaly detection

  • Authors:
  • Seungmin Lee;Gisung Kim;Sehun Kim

  • Affiliations:
  • Information Security Research Division, ETRI, 161, 138 Gajeongno, Yuseong-gu, Daejeon 305-700, South Korea;Internet Security Lab, Department of Industrial and Systems Engineering, School of Information Technologies, Korea Advanced Institute of Science and Technology, 291, Daehak-ro, Yuseong-Gu, Daejeon ...;Graduate School of Information Security, Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-Gu, Daejeon 305-701, South Korea

  • Venue:
  • Expert Systems with Applications: An International Journal
  • Year:
  • 2011

Quantified Score

Hi-index 12.05

Visualization

Abstract

As recent Internet threats are evolving more rapidly than ever before, one of the major challenges in designing an intrusion detection system is to provide early and accurate detection of emerging threats. In this study, a novel framework is developed for fully unsupervised training and online anomaly detection. The framework is designed so that an initial model is constructed and then it gradually evolves according to the current state of online data without any human intervention. In the framework, a self-organizing map (SOM) that is seamlessly combined with K-means clustering is transformed into an adaptive and dynamic algorithm suitable for real-time processing. The performance of the proposed approach is evaluated through experiments using the well-known KDD Cup 1999 data set and further experiments using the honeypot data recently collected from Kyoto University. It is shown that the proposed approach can significantly increase the detection rate while the false alarm rate remains low. In particular, it is capable of detecting new types of attacks at the earliest possible time.