Although the DNS over HTTPS (DoH) protocol has desirable properties for Internet users such as privacy and security, it also causes a problem in that network administrators are prevented from detecting suspicious network traffic generated by malware and malicious tools. To support their efforts in maintaining network security, in this paper, we propose a novel system that identifies malicious DNS tunnel tools through a hierarchical classification method that uses machine-learning technology on DoH traffic. We implemented a prototype of the proposed system and evaluated its performance on the CIRA-CIC-DoHBrw-2020 dataset, obtaining 99.81% accuracy in DoH traffic filtering, 99.99% accuracy in suspicious DoH traffic detection, and 97.22% accuracy in identification of malicious DNS tunnel tools.

DNS over HTTPS (DoH) protocol can mitigate the risk of privacy breaches but makes it difficult to control network security services due to the DNS traffic encryption. However, since malicious DNS tunnel tools for the DoH protocol pose network security threats, network administrators need to recognize malicious communications even after the DNS traffic encryption has become widespread. In this paper, we propose a malicious DNS tunnel tool recognition system using persistent DoH traffic analysis based on machine learning. The proposed system can accomplish continuous knowledge updates for emerging malicious DNS tunnel tools on the machine learning model. The system is based on hierarchical machine learning classification and focuses on DoH traffic analysis. The evaluation results confirm that the proposed system is able to recognize the six malicious DNS tunnel tools in total, not only well-known ones, including dns2tcp, dnscat2, and iodine, but also the emerging ones such as dnstt, tcp-over-dns, and tuns with 98.02% classification accuracy.