A Survey of Automatic Protocol Reverse Engineering Tools

Abstract

Computer network protocols define the rules in which two entities communicate over a network of unique hosts. Many protocol specifications are unknown, unavailable, or minimally documented, which prevents thorough analysis of the protocol for security purposes. For example, modern botnets often use undocumented and unique application-layer communication protocols to maintain command and control over numerous distributed hosts. Inferring the specification of closed protocols has numerous advantages, such as intelligent deep packet inspection, enhanced intrusion detection system algorithms for communications, and integration with legacy software packages. The multitude of closed protocols coupled with existing time-intensive reverse engineering methodologies has spawned investigation into automated approaches for reverse engineering of closed protocols. This article summarizes and organizes previously presented automatic protocol reverse engineering tools by approach. Approaches that focus on reverse engineering the finite state machine of a target protocol are separated from those that focus on reverse engineering the protocol format.

Inferring OpenVPN State Machines Using Protocol State Fuzzing

“…The idea of using regular inference to analyse implementations of security protocols dates back to at least Shu and Lee [19]. An extensive survey of this and other techniques to reverse engineer protocol implementations has been given by Narayan et al [20].…”

Experience Report: How to Extract Security Protocols' Specifications from C Libraries

“…This method is especially useful when a file to execute the protocol is unavailable because it only requires a system implementing the protocol to be running, even if it is not running locally. The work in [13] surveys Automatic Protocol Reverse Engineering Tools using this approach. Those tools seem to be mostly academic; a commercial one is VISUALETHER PROTOCOL ANALYZER 7.0 13 , which uses the output of Wireshark (a network protocol analyzer) to generate sequence and callflow diagrams.…”

“…The work in [13] surveys Automatic Protocol Reverse Engineering Tools using this approach. Those tools seem to be mostly academic; a commercial one is VISUALETHER PROTOCOL ANALYZER 7.0 13 , which uses the output of Wireshark (a network protocol analyzer) to generate sequence and callflow diagrams. Avalle et al [14] survey state-of-the-art research aimed at automatically getting formal security proofs of models close to the source code of real protocol-logic implementations.…”

Automatic Reverse Engineering of Private Flight Control Protocols of UAVs

“…Automatic protocol reverse-engineering [4] techniques enable private protocols to be understood. The goal of automatic protocol reverse engineering is to extract the grammar, semantics, and state machine of private protocol messages [5][6][7].…”

“…for item in SP do (4) for (Com , Com ) in Com do (5) if not equation (12) then (6) flag = 1 (7) break (8) end if (9) end for (10) if flag == 1 then (11) break (12) end if (13) join item in 2 (14) end for (15) The th bits in the check fields of commands Com and Com are assumed to be ( ) and ( ) , respectively, because they satisfy (9); ( ) and ( ) also satisfy…”