Normalizing Metamorphic Malware Using Term Rewriting

Abstract

Metamorphic malware -including certain viruses and worms -rewrite their code during propagation. This paper presents a method for normalizing multiple variants of metamorphic programs that perform their transformations using finite sets of instruction-sequence substitutions. The paper shows that the problem of constructing a normalizer can, in specific contexts, be formalized as a term rewriting problem. A general method is proposed for constructing normalizers. It involves modeling the metamorphic program's transformations as rewrite rules, and then modifying these rules to create a normalizing rule set. Casting the problem in terms of term rewriting exposes key challenges for constructing effective normalizers. In cases where the challenges cannot be met, approximations are proposed. The normalizer construction method is applied in a case study involving the virus called "W32.Evol". The results demonstrate that both the overall approach and the approximation schemes may have practical use on realistic malware, and may thus have the potential to improve signature-based malware scanners.

Android Mobile Platform Security and Malware Survey

“…Traditional static analysis approaches such as [38], [46], which focus on comparing programs with known malware based on the program code, looking for signatures using other heuristics. Other approaches [47], [48], [49] focus on using machine learning and data mining approaches for malware detection. In [49], Tesauro et al train a neural network to detect boot sector viruses, based on bytestring trigrams.…”

“…Other approaches [47], [48], [49] focus on using machine learning and data mining approaches for malware detection. In [49], Tesauro et al train a neural network to detect boot sector viruses, based on bytestring trigrams. Schultz et al [48] compare three machine learning algorithms trained on three features: DLL and system calls made by the program, strings found in the program binary, and a raw hexadecimal representation of the binary.…”

Exploiting an antivirus interface

“…More recently, metamorphic malware has appeared, which randomly applies binary transformations to its code segment during propagation in order to obfuscate features in the unencrypted portion. An example is the MetaPHOR system (c.f., [10]), which has become the basis for many other metamorphic malware propagation systems. Reversing these obfuscations to obtain reliable feature sets for signature-based detection is the subject of much current research [9,11,12], but case studies have shown that current antivirus detection schemes remain vulnerable to simple obfuscation attacks until the detector's signature database is updated to respond to the threat [13].…”

“…1 To defeat such a detector, we could have resorted to a more powerful metamorphic obfuscator such as the MetaPHOR system (c.f., [10]). MetaPHOR disassembles x86 binary code to a simplified intermediate language in which common sequences of instructions are expressed as single operations.…”

“…One obvious approach would be to apply existing polymorphic and metamorphic malware obfuscation engines such as MetaPHOR (c.f., [10]) to feature-containing portions of the malware using a succession of randomly chosen cryptographic keys and seed values until the unwanted features are removed. We intend to investigate such an approach in future work.…”