Security implications of memory deduplication in a virtualized environment

Using Hardware Features for Increased Debugging Transparency

“…In addition, many packers contain anti-debugging and anti-VM features, further increasing the challenge of reverseengineering packed malware. [12] API Call Kernel32!IsDebuggerPresent returns 1 if a target process is being debugged ntdll!NtQueryInformationProcess: ProcessInformation field set to -1 if the process is being debugged kernel32!CheckRemoteDebuggerPresent returns 1 in debugger process NtSetInformationThread with ThreadInformationClass set to 0x11 will detach some debuggers kernel32!DebugActiveProcess to prevent other debuggers from attaching to a process PEB Field PEB!IsDebugged is set by the system when a process is debugged PEB!NtGlobalFlags is set if the process was created by a debugger Detection ForceFlag field in heap header (+0x10) can be used to detect some debuggers UnhandledExceptionFilter calls a user-defined filter function, but terminates in a debugging process TEB of a debugged process contains a NULL pointer if no debugger is attached; valid pointer if some debuggers are attached Ctrl-C raises an exception in a debugged process, but the signal handler is called without debugging Inserting a Rogue INT3 opcode can masquerade as breakpoints Trap flag register manipulation to thwart tracers If entryPoint RVA is set to 0, the magic MZ value in PE files is erased ZwClose system call with invalid parameters can raise an exception in an attached debugger Direct context modification to confuse a debugger 0x2D interrupt causes debugged program to stop raising exceptions Some In-circuit Emulators (ICEs) can be detected by observing the behavior of the undocumented 0xF1 instruction Searching for 0xCC instructions in program memory to detect software breakpoints TLS-callback to perform checks [17] Invalid instruction behavior [14] Using memory deduplication to detect various hypervisors including VMware ESX server, Xen, and Linux KVM [56] Anti-emulation Bochs Visible debug port [10] QEMU cpuid returns less specific information [7] Accessing reserved MSR registers raises a General Protection (GP) exception in real hardware; QEMU does not [15] Attempting to execute an instruction longer than 15 bytes raises a GP exception in real hardware; QEMU does not [15] Undocumented icebp instruction hangs in QEMU [7], while real hardware raises an exception Unaligned memory references raise exceptions in real hardware; unsupported by QEMU [15] Bit 3 of FPU Control World register is always 1 in real hardware, while QEMU contains a 0 [7] Other Using CPU bugs or errata to create CPU fingerprints via public chipset documentation [15] To demonstrate the transparency of MALT, we use popular packing tools to pack the Notepad.exe application in a Windows environment and run this packed application in MALT with near return stepping mode, OllyDbg [31], DynamoRIO [57], and a Windows virtual machine, respectively. Ten packing tools are used, including UPX, Obsidium, ASPack, Armadillo, Themida, RLPack, PELock, VMProtect, eXPressor, and PECompact.…”

Undermining Isolation Through Covert Channels in the Fiasco.OC Microkernel

“…Suzaki et al [32] examined various virtual machines and discovered a side-channel in KVM based on the memory deduplication feature, which merges same-content memory pages, allowing for increased memory utilization. Based on these findings, Xiao et al [31] constructed a covert channel with bandwidths of up to 1000 bits/s yet with a large memory footprint of ∼ 400MB. Lipinski et al [50] drew on work done by Ristenpart et al [48] and improved the method of hard disk contention achieving a 1000 times higher steganographic bandwidth compared to the 0.1 bits/s by Ristenpart et al Okamura [28] et al examined the load-based covert channel on the Xen hypervisor.…”

“…Shared resources, such as caches [27], CPU [28], network subsystems [29], [30], or memory management [31], [32], are not only prevalent in all modern systems, but also run the risk of being the conduit for covert channel communication.…”

“…This is particularly notable as advanced system architectures employ hypervisors or other small isolation kernels with the goal of enforcing strict security policies. The memory deduplication feature in modern VMs is exploitable [31], [32], such that it allows to create very high-bandwidth channels. Lalande et al [35] prove that it is possible to form covert channels with various Linux subsystems, such as shared settings, system logs, thread enumeration, processor statistics, processor frequency, only to name a few.…”

Detection of Service Level Agreement (SLA) Violation in Memory Management in Virtual Machines

“…For example, many mainframe virtual machine hypervisors such as VMware ESX and ESXi [10], Extended Xen [11], and KSM (Kernel Samepage Merging) [12] of the Linux kernel use the technique of memory deduplication to reduce memory footprint size of virtual machines. Since previous research has shown that page level memory sharing could create a side channel for information leakage [13], [14], [15], many end users ask the hypervisor to disable memory deduplication for their VMs. However, there exists no solution for end users to verify the execution of this agreement other than trusting the words of the cloud provider.…”