A fast automaton-based method for detecting anomalous program behaviors

Abstract

Forrest et a1 introduced a new intrusion detection approach that ident$es anomalous sequences of system calls executed by programs. Since their work, anomaly detection on system call sequences has become perhaps the most successful approach for detecting novel intrusions. A natural way for learning sequences is to use a jinite-state automaton (FSA). However; previous research seemed to indicate that FSA-learning is coniputationally expensive, that it cannot be completely automated, or that the space usage of the FSA may be excessive. We present a new approach in this paper that overcomes these dificulties. Our approach buildsa compact FSA in a fully automaticand eficient manner; without requiring access to source code for programs. The space requirements for the FSA is low -of the order of a few kilobytes for typical programs. The FSA uses only a constant time per system call during the learning us well as detection period. This factor leads to low overheads for intrusion detection. Unlike many of the previous techniques, our FSA-technique can capture both short term and long term temporal relationships among system calls, and thus perform more accurute detection. For instance, the FSA can capture common program structures such as branches, joins, loops etc. This enables our approach to generalize and predict future behaviors from past behaviors. For instance, i f a program executed a loop once in an execution, the FSA approach can generalize and predict that the same loop may be executed zero or more times in subsequent executions. As a result, the training periods needed for our FSA based approach are shortel: Moreover; false positives are reduced without increasing the likelihood of missing attacks. This paper describes our FSA based technique and presents a comprehensive experimental evaluation of the technique.

Context Sensitive Anomaly Monitoring of Process Control Flow to Detect Mimicry Attacks and Impossible Paths

“…An important characteristic that attackers use is that the default protection model permits programs to invoke any system call from any function, but in actuality each system call is only invoked from a few locations in the legal code. While some previous work has exploited the idea of binding system calls or other security sensitive events with context [5,18,[22][23][24], this paper explores this approach further. We introduce the concept of waypoints to provide trustworthy control flow information, and show how to apply the information in anomaly detection.…”

“…The third layer of defense attempts to prevent the executing attack code from doing further harm though the system interface. Existing work at this stage includes anomaly detection [5,6,12,24,25,27], process randomization [2,10,13,28,36], and instruction set randomization [34,35].…”

“…Realizing that lack of context information in detection leads to certain false negatives possible (e.g., the impossible-path problem and the mimicry attacks), some anomaly monitors apply partial context information in anomaly detection [5,24,25]. The benefit of using context information is that control path information between two system call invocations can help detecting anomaly.…”

“…Context related permissions can help in this situation. [24] associates a system call with its invocation address. The return-into-lib(c) attack calls a library function, rather than a piece of shellcode.…”

Addressing Automated Adversaries of Network Applications

“…The branch of research most related to our approach is anomaly-based application integrity checking [16,34,50,99,119], which validates application behavior from the vantage point of a secure operating system. These approaches work well when an adversary has difficulty infiltrating the host system, however, they are inappropriate for the cheating problem where the adversary owns the machine and can readily alter the operating system to disable detection tools.…”