Shadow attacks: automatically evading system-call-behavior based malware detection

Abstract

Abstract. Contemporary malware makes extensive use of different techniques such as packing, code obfuscation, polymorphism, and metamorphism, to evade signature-based detection. Traditional signature-based detection technique is hard to catch up with latest malware or unknown malware. Behavior-based detection models are being investigated as a new methodology to defeat malware. This kind of approaches typically relies on system call sequences/graphs to model a malicious specification/pattern. In this paper, we present a new class of attacks, namely "shadow attacks", to evade current behaviorbased malware detectors by partitioning one piece of malware into multiple "shadow processes". None of the shadow processes contains a recognizable malicious behavior specification known to single-process-based malware detectors, yet those shadow processes as an ensemble can still fulfill the original malicious functionality. To demonstrate the feasibility of this attack, we have developed a compiler-level prototype tool, AutoShadow, to automatically generate shadow-process version of malware given the source code of original malware. Our preliminary result has demonstrated the effectiveness of shadow attacks in evading several behavior-based malware analysis/detection solutions in real world. With the increasing adoption of multi-core computers and multi-process programs, malware writers may exploit more such shadow attacks in the future. We hope our preliminary study can foster more discussion and research to improve current generation of behavior-based malware detectors to address this great potential threat before it becomes a security problem of the epidemic proportions.

Quantifiable Run-Time Kernel Attack Surface Reduction

“…Because behavioral systems do not make assumptions on the types of attacks that can be detected, they target detection of unknown attacks, unlike signature-based intrusion detection systems which can be easily bypassed by new attacks. It has also been shown that it is possible for attackers to bypass such detection mechanisms as well [30,38,53,55]. Hence, although behavioral intrusion detection could, as a side effect, reduce the kernel attack surface (because a kernel exploit's sequence of system calls might deviate from the normal use of the application), it is bypassable by using one of many known techniques, especially in the context of kernel attack surface reduction.…”

Vetting undesirable behaviors in android apps with permission use analysis

“…First, traditional solutions would only intercept a lot of system calls used to interact with the Binder driver, hiding the real actions performed by the application. Second, the use of IPC in Android apps breaks the execution flow of an app into chains among multiple processes, making the evasion of traditional syscall-based behavior monitoring easier [42].…”

Employing Program Semantics for Malware Detection

“…To counter system-call based approaches, malware authors make use of shadow attacks [13] and system-call injection attacks [14]. The feasibility of former attacks was first demonstrated by authors in [13].…”

“…The feasibility of former attacks was first demonstrated by authors in [13]. The authors in their paper have shown that the critical system-call sequences of malware can be divided and exported into separate shadow processes.…”