This paper presents a new method for steganography detection in network protocols. The method is based on a multilayer approach for the selective analysis of derived and aggregated metrics utilizing machine learning algorithms. The main objective is to provide steganalysis capability for networks with large numbers of devices and connections. We discuss considerations for performance analysis and present results. We also describe a means of applying our method for multilayer detection of a popular RSTEG (Retransmission Steganography) technique.

This paper demonstrates the effectiveness of using anomaly detection in cyclic communication as a method aimed at protecting industrial installations from steganographic communication and a wide range of cyberattacks. The analysis was performed for a method based on deterministic finite automaton and the authors’ method using cycles. In this paper, we discuss the cycle detection algorithm and graph construction as well as demonstrate an anomaly detection method for cyberattack detection that utilizes stochastic elements, such as time-to-response and time-between-messages. We present a novel algorithm that combines finite automaton determinism modeling consecutive admissible messages with a time-domain model allowing for random deviations of regularity. The study was conducted for several test scenarios, including C&C steganographic channels generated using the Modbus TCP/IP protocol. Experimental results demonstrating the effectiveness of the algorithms are presented for both methods. All algorithms described in this paper are implemented and run as part of a passive warden system embedded in a bigger commercial IDS (intrusion detection system).