An efficient method to detect periodic behavior in botnet traffic by analyzing control plane traffic

Abstract

Botnets are large networks of bots (compromised machines) that are under the control of a small number of bot masters. They pose a significant threat to Internet’s communications and applications. A botnet relies on command and control (C2) communications channels traffic between its members for its attack execution. C2 traffic occurs prior to any attack; hence, the detection of botnet’s C2 traffic enables the detection of members of the botnet before any real harm happens. We analyze C2 traffic and find that it exhibits a periodic behavior. This is due to the pre-programmed behavior of bots that check for updates to download them every T seconds. We exploit this periodic behavior to detect C2 traffic. The detection involves evaluating the periodogram of the monitored traffic. Then applying Walker’s large sample test to the periodogram’s maximum ordinate in order to determine if it is due to a periodic component or not. If the periodogram of the monitored traffic contains a periodic component, then it is highly likely that it is due to a bot’s C2 traffic. The test looks only at aggregate control plane traffic behavior, which makes it more scalable than techniques that involve deep packet inspection (DPI) or tracking the communication flows of different hosts. We apply the test to two types of botnet, tinyP2P and IRC that are generated by SLINGbot. We verify the periodic behavior of their C2 traffic and compare it to the results we get on real traffic that is obtained from a secured enterprise network. We further study the characteristics of the test in the presence of injected HTTP background traffic and the effect of the duty cycle on the periodic behavior.

BotCapturer: Detecting Botnets based on Two-Layered Analysis with Graph Anomaly Detection and Network Traffic Clustering

“…Our clustering model is based on C-flow and its four features [1]. A C-flow is a collection of all flows over a period of time.…”

“…But they can be ineffective when bot variants are encountered. As for behavior-based detection, AsSadhan et al [1] detected botnet by finding the periodic behavior from network traffic. The advantage of such technique is that it is based on a basic property shared by many botnet variants and is independent of the structure (e.g., centralized, P2P) and C&C protocol (e.g., IRC, HTTP).…”

“…Choi et al [4] detected botnet activities by monitoring group activities in DNS traffic, assuming that a bot must look up the domain names of the C&C server. Another representative approach is Botminer [1] that performed across clusters of communication traffic and malicious activity. Botminer is independent of botnet structure and C&C protocol.…”

Analyze Network Protocol's Hidden Behavior

“…The protocol hidden behavior analysis issue is given an undocumented protocol program C and captured protocol message M, monitoring and analy zing the process of C parsing M, (1) building the protocol behavior instruction sequences ; (2) inferring the potential trigger conditions according to the public behavior instruction sequences ; (3) mining and triggering the protocol's hidden behavior by static clustering analysis; (4) finally, evaluating the protocol's execution security. Our approach us es dynamic binary analysis combin ing static clustering techniques, which is based on the intuition that the most important protocol behavior is contained in the message parsing process.…”

“…Having mo re knowledge of netwo rk protocol is of great value in many network security applications, such as deep packet inspection [1] , botnet analysis [2] , vulnerability discovery [3] and signature generation [4] . Most protocol reverse analysis focus on analyzing and inferring unknown protocol specifications, such as message format and fields [5] , but pay litt le attention to the protocol's behavior.…”