Behavior-based malware detection

• Authors: Somesh Jha;Mihai Christodorescu
• Venue: Behavior-based malware detection
• Year: 2007

Malicious programs pose a significant problem in today's interconnected world. When malicious programs (malware) were few and far between, in the 1980s and 1990s, detection techniques based on syntactic descriptors (e.g., byte signatures) worked well. Analysts used their expertise to derive signatures for the latest malware outbreak and then these signatures were deployed to users of the detection software. The recent trend of malware for profit has stimulated the creativity of malware writers who now routinely use obfuscation and stealth to help their malware samples evade detection. As a result, we are witnessing a pandemic of malware infections that puts millions of victim machines under the control of attackers and that current detectors are unable to stem. This dissertation presents a series of behavior-based techniques for malware detection that are resilient to the obfuscation tactics exhibited by modern malware. The core element is a mechanism for specifying and detecting malicious behavior independently of any obfuscation artifacts. A new formalism, called malspecs, serves to characterize malicious behavior. A malspec consists of security-sensitive operations necessary to achieve a malicious goal and of constraints on the arguments of these operations, and is usually represented as a graph structure. Malspec detection builds on existing program analysis and verification techniques to achieve sound results. The key challenge is to develop semantic analyses that are oblivious to obfuscation yet achieve good runtime performance. Two techniques based on static analysis are presented in this dissertation to show how to balance detection power with detection overhead. To complement detection, a malspec mining algorithm is developed to extract malicious behavior from a known malware sample with no a priori information or assumptions. Observing that a malspec is the smallest sequence of operations necessary to achieve a malicious goal, the mining algorithm uses a differential dynamic analysis to construct malspecs automatically. Experimental evaluations of these techniques (both for malspec detection and malspec mining) on actual malware samples and benign programs indicate significantly improved detection rates, with only a limited performance impact. The results demonstrate that a principled behavioral analysis is fundamental to malware detection, superseding the inadequate syntactic detection schemes.