Probabilistic suffix models for API sequence analysis of Windows XP applications

  • Authors:
  • Geoffrey Mazeroff;Jens Gregor;Michael Thomason;Richard Ford

  • Affiliations:
  • Department of Computer Science, University of Tennessee Knoxville, 203 Claxton Complex, Knoxville, TN 37996-3450, USA;Department of Computer Science, University of Tennessee Knoxville, 203 Claxton Complex, Knoxville, TN 37996-3450, USA;Department of Computer Science, University of Tennessee Knoxville, 203 Claxton Complex, Knoxville, TN 37996-3450, USA;Department of Computer Sciences, Florida Institute of Technology, 150 W. University Blvd., Melbourne, FL 32901, USA

  • Venue:
  • Pattern Recognition
  • Year:
  • 2008

Quantified Score

Hi-index 0.01

Visualization

Abstract

Given the pervasive nature of malicious mobile code (viruses, worms, etc.), developing statistical/structural models of code execution is of considerable importance. We investigate using probabilistic suffix trees (PSTs) and associated suffix automata (PSAs) to build models of benign application behavior with the goal of subsequently being able to detect malicious applications as anything that deviates therefrom. We describe these probabilistic suffix models and present new generic analysis and manipulation algorithms. The models and the algorithms are then used in the context of API (i.e., system call) sequences realized by Windows XP applications. The analysis algorithms, when applied to traces (i.e., sequences of API calls) of benign and malicious applications, aid in choosing an appropriate modeling strategy in terms of distance metrics and consequently provide classification measures in terms of sequence-to-model matching. We give experimental results based on classification of unobserved traces of benign and malicious applications against a suffix model trained solely from traces generated by a small set of benign applications.