Enhancing the detection of metamorphic malware using call graphs

Elhadi, Ammar Ahmed E.; Maarof, Mohd Aizaini; Barry, Bazara I. A.; Hamza, Hentabli

doi:10.1016/j.cose.2014.07.004

Cited by 51 publications

(25 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Metamorphic malwares mutate and change their codes structure and signatures in each infection that is difficult to detect [1]. Lately, several host-based dynamical analysis techniques were proposed for metamorphic malware detection [2]. However, these techniques require separate environment to analyze malware in order to be able to be detected.…”

Section: Introductionmentioning

confidence: 99%

Metamorphic Malware Detection Based on Support Vector Machine Classification of Malware Sub-Signatures

et al. 2016

View full text Add to dashboard Cite

Achieving accurate and efficient metamorphic malware detection remains a challenge. Metamorphic malware is able to mutate and alter its code structure in each infection that can circumvent signature matching detection. However, some vital functionalities and code segments remain unchanged between mutations. We exploit these unchanged features by the mean of classification using Support Vector Machine (SVM). N-gram features are extracted directly from malware binaries to avoid disassembly, which these features are then masked with the extracted known malware signature n-grams. These masked features reduce the number of selected n-gram features considerably. Our method is capable to accurately detect metamorphic malware with~99% accuracy and low false positive rate. The proposed method is also superior to commercially available anti-viruses for detecting metamorphic malware. Keyword: SVM classification, Metamorphic, n-gram, SnortCopyright © 2016 Universitas Ahmad Dahlan. All rights reserved. IntroductionMalware is one of security attacks to Internet users as it breaches computer security and data confidentiality, which are categorized into general (non-mutable) and mutable types. Antivirus softwares rely on signature-based detection as the primary detection mechanism. Mutatable malware such as packing, polymorphic, and metamorphic make the detection based on signature matching difficult. Metamorphic malwares mutate and change their codes structure and signatures in each infection that is difficult to detect [1]. Lately, several host-based dynamical analysis techniques were proposed for metamorphic malware detection [2]. However, these techniques require separate environment to analyze malware in order to be able to be detected. At the same time, the requirement of binary code disassembly in opcode-based methods [3][4][5] is not suitable for timely metamorphic detection on host-level intrusion detection systems.We propose metamorphic malware detection based on static analysis of metamorphic malware binaries without disassembly. Features are extracted from binary, which can be in the form of packets payload in network detection system or files in host based detection system using n-gram feature extraction and machine learning SVM classification. Besides, extracted ngram features are masked with known malware signature n-grams to represent only informative malware features. This technique can reduce the n-gram search space. This paper is organized as follows. Section 2 provides a critical review on relevant literatures. The methodology for metamorphic malware detection in network and host-based IDS are described in Section 3. Section 4 highlights the experimental setup, datasets, and evaluation criteria. The data analysis and comparison with commercial anti-virus software are presented in Section 5. Section 6 concludes the research findings, and contributions of the paper.

show abstract

Section: Introductionmentioning

confidence: 99%

Metamorphic Malware Detection Based on Support Vector Machine Classification of Malware Sub-Signatures

et al. 2016

View full text Add to dashboard Cite

show abstract

“…1(c). In the literature, external call graphs have been used for malware detection (Elhadi et al, 2014). In such a case, model graphs and data graphs are compared in order to distinguish call graphs representing benign programs from those based on malware samples (Riesen et al, 2010;Elhadi et al, 2014).…”

Section: Function Call Graphmentioning

confidence: 99%

SIGMA: A Semantic Integrated Graph Matching Approach for identifying reused functions in binary code

Alrabaee

Shirani

Wang

et al. 2015

Digital Investigation

View full text Add to dashboard Cite

a b s t r a c tThe capability of efficiently recognizing reused functions for binary code is critical to many digital forensics tasks, especially considering the fact that many modern malware typically contain a significant amount of functions borrowed from open source software packages. Such a capability will not only improve the efficiency of reverse engineering, but also reduce the odds of common libraries leading to false correlations between unrelated code bases. In this paper, we propose SIGMA, a technique for identifying reused functions in binary code by matching traces of a novel representation of binary code, namely, the Semantic Integrated Graph (SIG). The SIG s enhance and merge several existing concepts from classic program analysis, including control flow graph, register flow graph, and function call graph into a joint data structure. Such a comprehensive representation allows us to capture different semantic descriptors of common functionalities in a unified manner as graph traces, which can be extracted from binaries and matched to identify reused functions, actions, or open source software packages. Experimental results show that our approach yields promising results. Furthermore, we demonstrate the effectiveness of our approach through a case study using two malware known to share common functionalities, namely, Zeus and Citadel.

show abstract

“…They detected malware by decision trees. Elhadi et al [21] proposed a method to build call graphs of programs at runtime and distinguish malware and benign programs by comparing their call graphs with known malware call graphs. Shehata et al [22] presented a detection method based on observing system calls at runtime.…”

Section: Dynamic Analysis-based Methodsmentioning

confidence: 99%

HM3alD: Polymorphic Malware Detection Using Program Behavior-Aware Hidden Markov Model

Tajoddin

Jalili

2018

Applied Sciences

View full text Add to dashboard Cite

Abstract:Malware have been tremendously growing in recent years. Most malware use obfuscation techniques for evasion and hiding purposes, but they preserve the functionality and malicious behavior of original code. Although most research work has been mainly focused on program static analysis, some recent contributions have used program behavior analysis to detect malware at run-time. Extracting the behavior of polymorphic malware is one of the major issues that affects the detection result. In this paper, we propose HM 3 alD, a novel program behavior-aware hidden Markov model for polymorphic malware detection. The main idea is to use an effective clustering scheme to partition the program behavior of malware instances and then apply a novel hidden Markov model (called program behavior-aware HMM) on each cluster to train the corresponding behavior. Low-level program behavior, OS-level system call sequence, is mapped to high-level action sequence and used as transition triggers across states in program behavior-aware HMM topology. Experimental results show that HM 3 alD outperforms all current dynamic and static malware detection methods, especially in term of FAR, while using a large dataset of 6349 malware.

show abstract

Enhancing the detection of metamorphic malware using call graphs

Cited by 51 publications

References 22 publications

Metamorphic Malware Detection Based on Support Vector Machine Classification of Malware Sub-Signatures

Metamorphic Malware Detection Based on Support Vector Machine Classification of Malware Sub-Signatures

SIGMA: A Semantic Integrated Graph Matching Approach for identifying reused functions in binary code

HM3alD: Polymorphic Malware Detection Using Program Behavior-Aware Hidden Markov Model

Contact Info

Product

Resources

About