An unsupervised anomaly detection framework for multiple-connection based network intrusions

• Authors: Wei Lu
• Venue: An unsupervised anomaly detection framework for multiple-connection based network intrusions
• Year: 2006

In this dissertation, we propose an effective and efficient online unsupervised anomaly detection framework. The framework consists of new anomalousness metrics, named IP Weight, and a new hybrid clustering algorithm, named I-means. IP Weight metrics provide measures of anomalousness of IP packet flows on networks. A simple classification of network intrusions consists of distinguishing between single-connection based attacks and multiple-connection based attacks. The IP weight metrics proposed in this work characterize specifically multiple-connection based attacks. The definition of specific metrics for single-connection based attacks is left for future work. The I-means algorithm combines mixture resolving, a genetic algorithm automatically estimating the optimal number of clusters for a set of data, and the k-means algorithm for clustering. Three sets of experiments are conducted to evaluate our new unsupervised anomaly detection framework. The first experiment empirically validates that IP Weight metrics reduce dimensions of feature space characterizing IP packets at a level comparable with the principal component analysis technique. The second experiment is an offline evaluation based on 1998 DARPA intrusion detection dataset. In the offline evaluation, we compare our framework with three other unsupervised anomaly detection approaches, namely, plain k-means clustering, univariate outlier detection and multivariate outlier detection. Evaluation results show that the detection framework based on I-means yields the highest detection rate with a low false alarm rate. Specifically, it detects 18 types of attacks out of a total of 19 multiple-connection based attack types. The third experiment is an online evaluation in a live networking environment. The evaluation result not only confirms the detection effectiveness observed with the DARPA dataset, but also shows a good runtime efficiency, with response times falling within few seconds ranges.