Stealthy malware detection through vmm-based "out-of-the-box" semantic view reconstruction

  • Authors:

  • Xuxian Jiang;Xinyuan Wang;Dongyan Xu

  • Affiliations:

  • George Mason University, Fairfax, VA;George Mason University, Fairfax, VA;Purdue University, West Lafayette, IN

  • Venue:
  • Proceedings of the 14th ACM conference on Computer and communications security
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

An alarming trend in malware attacks is that they are armed with stealthy techniques to detect, evade, and subvert malware detection facilities of the victim. On the defensive side, a fundamental limitation of traditional host-based anti-malware systems is that they run inside the very hosts they are protecting ("in the box"), making them vulnerable to counter-detection and subversion by malware. To address this limitation, recent solutions based on virtual machine (VM) technologies advocate placing the malware detection facilities outside of the protected VM ("out of the box"). However, they gain tamper resistance at the cost of losing the native, semantic view of the host which is enjoyed by the "in the box" approach, thus leading to a technical challenge known as the semantic gap. In this paper, we present the design, implementation, and evaluation of VMwatcher - an "out-of-the-box" approach that overcomes the semantic gap challenge. A new technique called guest view casting is developed to systematically reconstruct internal semantic views (e.g., files, processes, and kernel modules) of a VM from the outside in a non-intrusive manner. Specifically, the new technique casts semantic definitions of guest OS data structures and functions on virtual machine monitor (VMM)-level VM states, so that the semantic view can be reconstructed. With the semantic gap bridged, we identify two unique malware detection capabilities: (1) view comparison-based malware detection and its demonstration in rootkit detection and (2) "out-of-the-box" deployment of host-based anti-malware software with improved detection accuracy and tamper-resistance. We have implemented a proof-of-concept prototype on both Linux and Windows platforms and our experimental results with real-world malware, including elusive kernel-level rootkits, demonstrate its practicality and effectiveness.