Protecting Kernel Code and Data with a Virtualization-Aware Collaborative Operating System

Abstract

The traditional virtual machine usage model advocates placing security mechanisms in a trusted VM layer and letting the untrusted guest OS run unaware of the presence of virtualization. In this work we challenge this traditional model and propose a collaboration approach between a virtualizationaware operating system and a VM layer to prevent tampering against kernel code and data. Our integrity model is a relaxed version of Biba's and the main idea is to have all attempted writes into kernel code and data segments checked for validity at VM level. The OS-VM collaboration bridges the semantic gap between tracing low integrity objects at OS-level (files, processes, modules, allocated areas) and architecture-level (memory and registers). We have implemented this approach in a proofof-concept prototype and have successfully tested it against 6 rootkits (including a non-control data attack) and 4 realworld benign LKM/drivers. All rootkits were prevented from corrupting kernel space and no false positive was triggered for benign modules. Performance measurements show that the average overhead to the VM for the OS-VM communication is low (7%, CPU benchmarks). The greatest overhead is caused by the memory monitoring module inside the VM: 1.38X alone and 1.46X when combined with the OS-VM communication. For OS microbenchmarks the slowdown for the OS-VM communication was 1.16X on average.

Bridging the Semantic Gap to Mitigate Kernel-Level Keyloggers

“…OSck [28] will treat our malicious function as benign as it has the same signature as the benign one. The strategy presented in [34] can defeat this keylogger as the hijack of the original receive buffer function will correspond to a write into the kernel data segment by a suspicious instruction (its bytes came from the network).…”

“…In previous work [34] we developed a proof-of-concept prototype for a system where a guest OS and a VM layer communicated to prevent tampering against kernel code and data segments at the architectural level. The system employed a dynamic information flow tracking (DIFT) system [9] that tainted network bytes.…”

A Socially-Aware Operating System for Trustworthy Computing

“…However we still believe that such assumption could provide coverage for a great number of attacks. We plan to adopt the same DIFT system we introduced in [31].…”

Guardian: Hypervisor as Security Foothold for Personal Computers

“…Many hypervisor-based security systems have been designed and reported in the literature. For instance, a hypervisor can be applied for I/O related protection [9,31], for kernel integrity protection [3,13,23,26,28,41,42], and for user space protection [6,7,12,21,34,43]. By studying these systems, we identify cryptographic engine, measurement, emulation, interception and manipulation as the fundamental security primitives which are adopted in Guardian as well.…”