Emulation-based detection of non-self-contained polymorphic shellcode

  • Authors:
  • Michalis Polychronakis;Kostas G. Anagnostakis;Evangelos P. Markatos

  • Affiliations:
  • Institute of Computer Science, Foundation for Research & Technology - Hellas;Institute for Infocomm Research, Singapore;Institute of Computer Science, Foundation for Research & Technology - Hellas

  • Venue:
  • RAID'07 Proceedings of the 10th international conference on Recent advances in intrusion detection
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

Network-level emulation has recently been proposed as a method for the accurate detection of previously unknown polymorphic code injection attacks. In this paper, we extend network-level emulation along two lines. First, we present an improved execution behavior heuristic that enables the detection of a certain class of non-self-contained polymorphic shellcodes that are currently missed by existing emulation-based approaches. Second, we present two generic algorithmic optimizations that improve the runtime performance of the detector. We have implemented a prototype of the proposed technique and evaluated it using off-the-shelf non-self-contained polymorphic shellcode engines and benign data. The detector achieves a modest processing throughput, which however is enough for decent runtime performance on actual deployments, while it has not produced any false positives. Finally, we report attack activity statistics from a seven-month deployment of our prototype in a production network, which demonstrate the effectiveness and practicality of our approach.