Hybrid engine for polymorphic shellcode detection

  • Authors:
  • Udo Payer;Peter Teufl;Mario Lamberger

  • Affiliations:
  • Institute of Applied Information Processing and Communications, Graz, Austria;Institute of Applied Information Processing and Communications, Graz, Austria;Institute of Applied Information Processing and Communications, Graz, Austria

  • Venue:
  • DIMVA'05 Proceedings of the Second international conference on Detection of Intrusions and Malware, and Vulnerability Assessment
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

Driven by the permanent search for reliable anomaly-based intrusion detection mechanisms, we investigated different options of neural network (NN) based techniques. A further improvement could be achieved by combining the best suited NN-based data mining techniques with a mechanism we call “execution chain evaluation”. This means that disassembled instruction chains are processed by the NN in order to detect malicious code. The proposed detection engine was trained and tested in various ways. Examples were taken from all publicly available polymorphic shellcode engines as well as from self-designed engines. A prototype implementation of our sensor has been realized and integrated as a plug-in into the SNORTTM[13] intrusion detection system.