Operating system interface obfuscation and the revealing of hidden operations

  • Authors:
  • Abhinav Srivastava;Andrea Lanzi;Jonathon Giffin;Davide Balzarotti

  • Affiliations:
  • School of Computer Science, Georgia Institute of Technology;Institute Eurecom;School of Computer Science, Georgia Institute of Technology;Institute Eurecom

  • Venue:
  • DIMVA'11 Proceedings of the 8th international conference on Detection of intrusions and malware, and vulnerability assessment
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Many software security solutions--including malware analyzers, information flow tracking systems, auditing utilities, and host-based intrusion detectors--rely on knowledge of standard system call interfaces to reason about process execution behavior. In this work, we show how a rootkit can obfuscate a commodity kernel's system call interfaces to degrade the effectiveness of these tools. Our attack, called Illusion, allows user-level malware to invoke privileged kernel operations without requiring the malware to call the actual system calls corresponding to the operations. The Illusion interface hides system operations from user-, kernel-, and hypervisor-level monitors mediating the conventional system-call interface. Illusion alters neither static kernel code nor read-only dispatch tables, remaining elusive from tools protecting kernel memory. We then consider the problem of Illusion attacks and augment system call data with kernel-level execution information to expose the hidden kernel operations. We present a Xen-based monitoring system, Sherlock, that adds kernel execution watchpoints to the stream of system calls. Sherlock automatically adapts its sensitivity based on security requirements to remain performant on desktop systems: in normal execution, it adds 1% to 10% overhead to a variety of workloads.