Fast and memory-efficient regular expression matching for deep packet inspection

Abstract

Packet content scanning at high speed has become extremely important due to its applications in network security, network monitoring, HTTP load balancing, etc. In content scanning, the packet payload is compared against a set of patterns specified as regular expressions. In this paper, we first show that memory requirements using traditional methods are prohibitively high for many patterns used in packet scanning applications. We then propose regular expression rewrite techniques that can effectively reduce memory usage. Further, we develop a grouping scheme that can strategically compile a set of regular expressions into several engines, resulting in remarkable improvement of regular expression matching speed without much increase in memory usage. We implement a new DFA-based packet scanner using the above techniques. Our experimental results using real-world traffic and patterns show that our implementation achieves a factor of 12 to 42 performance improvement over a commonly used DFAbased scanner. Compared to the state-of-art NFA-based implementation, our DFA-based packet scanner achieves 50 to 700 times speedup.

Optimizing Regular Expression Matching with SR-NFA on Multi-Core Systems

“…For each input character, the DFA makes one access to the state transition table (STT), performs a constant amount of computation, and finds a single transition target. While being computationally efficient (and optimal), the DFA can suffer from state explosion [12,22,27] during its construction, where the number of states required by the DFA is quadratically or exponentially larger than the original NFA [17,27]. State explosion can be caused by certain regex patterns, or by combining several non-exploding patterns in one DFA [27].…”

“…For some regexes, converting the O(n)-state NFA to a DFA can generate an excessively large STT with up to O(2 n ) states and O(2 n |Σ|) transitions, a phenomenon known as the exponential state explosion [17,27]. In these cases, DFA-based REM becomes impractical on multi-core systems for several reasons.…”

“…Alternatively, the regex can be converted to a deterministic finite automaton (DFA) for efficient matching, which however may require an extremely large state transition table (STT) due to exponential state explosion [17,27]. We propose the segmented regex-NFA (SR-NFA) architecture, where the regex is first compiled into modular nondeterministic finite automata (NFA), then partitioned, optimized, and matched efficiently on modern multi-core processors.…”

Fast and Memory-Efficient Regular Expression Matching Using Transition Sharing

“…In [6] Fang Yu et al systematically analyze the regular expressions which are commonly used in networking applications. They analyze individual regular expressions that will lead DFAs with quadratic size or exponential size, and show that traditional methods are prohibitively high for patterns used in packet scanning applications.…”

“…However, memory requirements are high in practical deep packet inspection. For practical networking applications, since we don't have any prior knowledge of whether/where a matching substring may appear, most applications compile the rules together into a single one-pass search mode DFA [6] to achieve O (1) processing complexity. This means ".…”