On evolving buffer overflow attacks using genetic programming

  • Authors:
  • Hilmi Güneş Kayacik;Malcolm Heywood;Nur Zincir-Heywood

  • Affiliations:
  • Dalhousie University, Halifax, Nova Scotia, Canada;Dalhousie University, Halifax, Nova Scotia, Canada;Dalhousie University, Halifax, Nova Scotia, Canada

  • Venue:
  • Proceedings of the 8th annual conference on Genetic and evolutionary computation
  • Year:
  • 2006

Quantified Score

Hi-index 0.01

Visualization

Abstract

In this work, we employed genetic programming to evolve a "white hat" attacker; that is to say, we evolve variants of an attack with the objective of providing better detectors. Assuming a generic buffer overflow exploit, we evolve variants of the generic attack, with the objective of evading detection by signature-based methods. To do so, we pay particular attention to the formulation of an appropriate fitness function and partnering instruction set. Moreover, by making use of the intron behavior inherent in the genetic programming paradigm, we are able to explicitly obfuscate the true intent of the code. All the resulting attacks defeat the widely used 'Snort' Intrusion Detection System.