Botnets have become one of the major threats on the Internet for serving as a vector for carrying attacks against organizations and committing cybercrimes. They are used to generate spam, carry out DDOS attacks and click-fraud, and steal sensitive information. In this paper, we propose a new approach for characterizing and detecting botnets using network traffic behaviors. Our approach focuses on detecting the bots before they launch their attack. We focus in this paper on detecting P2P bots, which represent the newest and most challenging types of botnets currently available. We study the ability of five different commonly used machine learning techniques to meet on line botnet detection requirements, namely adaptability, novelty detection, and early detection. The results of our experimental evaluation based on existing datasets show that it is possible to detect effectively botnets during the botnet Command-and Control (C&C) phase and before they launch their attacks using traffic behaviors only. However, none of the studied techniques can address all the above requirements at once.

No abstract

I would like to express my profound gratitude and appreciation to my advisor, Prof. Lee Bu Sung, Francis, for providing invaluable wisdom and guidance during this research. His broad knowledge and deep insights helped me to choose the correct methodology to carry out this research. Special thanks and gratitude go to my co-supervisor, Prof. Amitabha Das for giving me invaluable advice every time I needed. He helped me to choose this wonderful research topic and pursue passionately. I hope and look forward to continuous collaboration with Prof. Lee and Prof. Das in the future. I am especially grateful to Dr. Seet Boon Chong, whose close cooperation with me enabled me to solve many details of this research. Thanks to Prof. Dusit Niyato for the technical discussions about evolutionary game theory and its applications in our research. Their enthusiastic participation helped me solve numerous problems during my research.