Polymorphic Code Detection with GA Optimized Markov Models

Abstract

Abstract. This paper presents our progression in the search for reliable anomaly-based intrusion detection mechanisms. We investigated different options of stochastic techniques. We started our investigations with Markov chains to detect abnormal traffic. The main aspect in our prior work was the optimization of transition matrices to obtain better detection accuracy. First, we tried to automatically train the transition matrix with normal traffic. Then, this transition matrix was used to calculate the probabilities of a dedicated Markov sequence. This transition matrix was used to find differences between the trained normal traffic and characteristic parts of a polymorphic shellcode. To improve the efficiency of this automatically trained transition matrix, we modified some entries in a way that byte-sequences of typical shellcodes substantially differs from normal network behavior. But this approach did not meet our requirements concerning generalization. Therefore we searched for automatic methods to improve the matrix. Genetic algorithms are adequate tools if just little knowledge about the search space is available and the complexity of the problem is very hard (NP-complete).

Specification and evaluation of polymorphic shellcode properties using a new temporal logic

“…Note that, Clet engine is endowed with a spectrum analysis mechanism which was designed in order to defeat Data Mining methods. However, evaluation results obtained in [36,37] show a detection rate of 100% for Clet engine with a low rate of false positives.…”

“…In these works, Data Mining methods are used as a learning process which is performed over a set of samples (positives and negatives datasets). For instance, in [37] authors suggest the use of Neural Networks as training process, whereas in [36] authors propose to use the Markov Chains. Note that, Clet engine is endowed with a spectrum analysis mechanism which was designed in order to defeat Data Mining methods.…”

“…Regarding anomaly-based approaches, there is only few works [36,37] that focus mainly on polymorphic shellcodes. In these works, Data Mining methods are used as a learning process which is performed over a set of samples (positives and negatives datasets).…”

A Hybrid Detection Approach for Zero-Day Polymorphic Shellcodes

“…This approach is robust but exhausted because of disassemble. Udo Payer et al [9] give a similar approach to detect executable code in flows, but use Markov Mode to bytes rather than instructions, so it achieves a better performance, but it cannot distinguish shellcode from benign executable code.…”

“…According to [9], the probability of occurrence of a byte sequence is evidently different between data and executable codes. We use Marcov Mode [9, 10] to profile the difference.…”