Proceedings of the 6th International Conference on Security of Information and Networks
| Hi-index | 0.00 |
Software attacks are a serious problem. Conventional anti-malware software expects malicious software, malware, to contain fixed and known code. Malware writers have devised methods of concealing or constantly changing their attacks to evade anti-malware software. Two important recent techniques are polymorphism, which makes uses of code encryption, and metamorphism, which uses a variety of code obfuscation techniques. This dissertation presents three new techniques for detection of these malware. The first technique is to recognize polymorphic malware that are encrypted and that self-decrypt before launching the attacks in network traffic. We propose a new approach that combines static analysis and instruction emulation techniques to more accurately identify the starting location and instructions of the decryption routine, which is characteristic of such malware, even if self-modifying code is used. This method has been implemented and tested on current polymorphic exploits, including ones generated by state-of-the-art polymorphic engines. All exploits have been detected (i.e., a 100% detection rate), including those for which the decryption routine is dynamically coded or self-modifying. The method has also been tested on benign network traffic and Windows executables. The false positive rates are approximately .0002% and .01% for these two categories, respectively. Running time is approximately linear in the size of the network payload being analyzed and is between 1 and 2 MB/s.