Effective dimension in anomaly detection: its application to computer systems

  • Authors:
  • Tsuyoshi Idé;Hisashi Kashima

  • Affiliations:
  • IBM Research, Tokyo Research Laboratory, Yamato-shi, Kanagawa, Japan;IBM Research, Tokyo Research Laboratory, Yamato-shi, Kanagawa, Japan

  • Venue:
  • JSAI'03/JSAI04 Proceedings of the 2003 and 2004 international conference on New frontiers in artificial intelligence
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

We consider the issue of online anomaly detection from a time sequence of directional data (normalized vectors) in high dimensional systems. In spite of the practical importance, little is known about anomaly detection methods for directional data. Using a novel concept of the effective dimension of the system, we successfully formulated an anomaly detection method which is free from the "curse of dimensionality." In our method, we derive a probability distribution function (pdf) for an anomaly metric, and use a novel update algorithm for the parameters in the pdf, where the effective dimension is included as a fitting parameter. For directional data from a computer system, we demonstrate the utility of our algorithm in anomaly detection.