Network–Level polymorphic shellcode detection using emulation

  • Authors:
  • Michalis Polychronakis;Kostas G. Anagnostakis;Evangelos P. Markatos

  • Affiliations:
  • Institute of Computer Science, Foundation for Research & Technology –– Hellas;Institute for Infocomm Research, Singapore;Institute of Computer Science, Foundation for Research & Technology –– Hellas

  • Venue:
  • DIMVA'06 Proceedings of the Third international conference on Detection of Intrusions and Malware & Vulnerability Assessment
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

As state–of–the–art attack detection technology becomes more prevalent, attackers are likely to evolve, employing techniques such as polymorphism and metamorphism to evade detection. Although recent results have been promising, most existing proposals can be defeated using only minor enhancements to the attack vector. We present a heuristic detection method that scans network traffic streams for the presence of polymorphic shellcode. Our approach relies on a NIDS–embedded CPU emulator that executes every potential instruction sequence, aiming to identify the execution behavior of polymorphic shellcodes. Our analysis demonstrates that the proposed approach is more robust to obfuscation techniques like self-modifications compared to previous proposals, but also highlights advanced evasion techniques that need to be more closely examined towards a satisfactory solution to the polymorphic shellcode detection problem