Comprehensive shellcode detection using runtime heuristics

Abstract

A promising method for the detection of previously unknown code injection attacks is the identification of the shellcode that is part of the attack vector using payload execution. Existing systems based on this approach rely on the self-decrypting behavior of polymorphic code and can identify only that particular class of shellcode. Plain, and more importantly, metamorphic shellcode do not carry a decryption routine nor exhibit any self-modifications and thus both evade existing detection systems. In this paper, we present a comprehensive shellcode detection technique that uses a set of runtime heuristics to identify the presence of shellcode in arbitrary data streams. We have identified fundamental machine-level operations that are inescapably performed by different shellcode types, based on which we have designed heuristics that enable the detection of plain and metamorphic shellcode regardless of the use of self-decryption. We have implemented our technique in Gene, a code injection attack detection system based on passive network monitoring. Our experimental evaluation and real-world deployment show that Gene can effectively detect a large and diverse set of shellcode samples that are currently missed by existing detectors, while so far it has not generated any false positives.

Efficient Shellcode Detection on Commodity Hardware

“…In other words, our system tries each byte of the network stream as a potential entry point of shellcode. Similar to the dynamic approaches [1], [6], [9], we refer to a complete execution from each offset of the stream as an execution chain. Compared with the dynamic approaches that require intercepting and tracking each instruction, our approach allows instruction sequences being executed directly on the CPU, and only trap a small number of instructions selectively by utilizing hardware paging mechanisms.…”

“…In order to detect shellcode in our execution environment, we employ the PEB heuristic [6]. This detection heuristic relies on the conventional behaviour of shellcode: After executing the network string, the shellcode will first read the memory address of FS:[0x30] to locate the PEB, and then read the LoaderData field (0xc bytes offset) of the PEB to locate the PEB LDR DATA structure.…”

“…When we utilize 2 CPU cores for two different detection processes, our system can achieve 73.8 Mbps network throughput without packet loss. Compared with the emulation-based methods [1], [6], [9], the network throughput is greatly improved.…”

“…In the past few years, many detection approaches [1], [3], [4], [6], [9] have been proposed. Basically, these methods can be divided into two categories: static analysis [3], [4] and dynamic analysis [1], [6], [9].…”

“…Basically, these methods can be divided into two categories: static analysis [3], [4] and dynamic analysis [1], [6], [9]. The core idea of static analysis is to disassemble the network stream and then analyze the code-level patterns that could be signatures obtained from existing shellcode.…”