Discovering Similarities in Malware Behaviors by Clustering of API Call Sequences

A Real-Time and Adaptive-Learning Malware Detection Method Based on API-Pair Graph

“…In our model, an API pair would be adopted as meta for detection. The idea was based on the thought of Markov chain and has been evaluated in [11], [12], which defined each type of event as a system state, and the state was represented as a graph to describe the state of change [13]. When some events were executed in a specific order, it may clarify that some abnormal behaviors occurred.…”

Semi-Supervised Malware Clustering Based on the Weight of Bytecode and API

“…Dynamic analysis has the advantage of generating accurate analysis results regardless of the level of anti-static analysis techniques used in the malware, such as code obfuscation, encryption. Thus, dynamic features have been applied to malware clustering in some existing work [10], [11], [21], [27], [28]. Perdisci et al [11] presented a new scalable system for network-level behavioral clustering of HTTP-based malware that aims to efficiently group newly collected malware samples into malware family clusters.…”

“…The sequences of API calls are dealt with as text-like terms and terms are constructed with N-grams for the clustering process. Shamsi et al [10] proposed a hierarchical clustering algorithm based on the API call sequences. By comparing the accuracy of Optimal Matching (OM), the Longest Common Subsequence (LCS), and Long Common Prefix (LCP) [29], [30] for calculating the distance of sequences, the LCP is chosen as the sequence comparison algorithm for hierarchical clustering.…”

“…Static analysis is simple and fast, but it can be easily evaded if malware is packed or obfuscated. Hence, more researchers focused on the clustering approaches based on dynamic analysis [10]- [13]. These approaches are less vulnerable to mutation schemes, such as run-time packers or binary obfuscation.…”

“…Therefore, it is necessary to try new similarity measurement methods, with excellent performance in other fields for clustering. For the clustering algorithm, there are many clustering algorithms applied for malware detection, such as hierarchical clustering algorithms [10], [11], density-based clustering algorithms [5], [18], and prototype-based clustering algorithms [4], [19]. However, these hierarchical clustering algorithms are sensitive to noise and outliers, and they cannot handle clusters of different sizes and convex shaped clusters, either.…”