Hybrid analysis and control of malicious code

• Authors: Barton P. Miller;Kevin A. Roundy
• Venue: Hybrid analysis and control of malicious code
• Year: 2012

State of the art analysis techniques for malicious executables lag significantly behind their counterparts for compiler-generated executables. This difference exists because 90% of malicious software (also known as malware) actively resists analysis. In particular, most malware resists static attempts to recover structural information from its binary code, and resists dynamic attempts to observe and modify its code.In this dissertation, we develop static and dynamic techniques and combine them in a hybrid algorithm that preserves the respective strengths of these techniques while mitigating their weaknesses. In particular, we build structural analyses with static parsing techniques that can disassemble arbitrarily obfuscated binary code with high accuracy, and recover the structure of that code in terms of functions, loops, and basic blocks. We develop dynamic techniques to identify transitions into statically unreachable code and respond to malware that overwrites its code. These dynamic techniques remove overwritten and unreachable code from our analysis and trigger additional parsing at entry points into un-analyzed code, before this code executes. Our stealthy instrumentation techniques leverage our structural analysis to stealthily and efficiently instrument binary code that resists modification. These instrumentation techniques hide the modifications they make to the binary code, and the additional space that they allocate in the program's address space to hold instrumentation. We demonstrate the utility of our techniques by adapting the Dyninst 7.0 binary analysis and instrumentation tool so that its users can analyze defensive malware code in exactly the same way that they analyze non-defensive binaries. We also build customizable malware analysis factories that perform batch-processing of malware binaries in an isolated environment, to help security companies efficiently process the tens of thousands of new malware samples that they receive each day. Finally, we use our analysis factory to study the most prevalent defensive techniques used by malware binaries. We thereby provide a snapshot of the obfuscation techniques that we have seen to date, and demonstrate that our techniques allow us to analyze and instrument highly defensive binary code.