On the complexity and performance of parsing with derivatives

Abstract

Current algorithms for context-free parsing inflict a tradeoff between ease of understanding, ease of implementation, theoretical complexity, and practical performance. No algorithm achieves all of these properties simultaneously. Might et al. introduced parsing with derivatives, which handles arbitrary context-free grammars while being both easy to understand and simple to implement. Despite much initial enthusiasm and a multitude of independent implementations, its worst-case complexity has never been proven to be better than exponential. In fact, high-level arguments claiming it is fundamentally exponential have been advanced and even accepted as part of the folklore. Performance ended up being sluggish in practice, and this sluggishness was taken as informal evidence of exponentiality. In this paper, we reexamine the performance of parsing with derivatives. We have discovered that it is not exponential but, in fact, cubic. Moreover, simple (though perhaps not obvious) modifications to the implementation by Might et al. lead to an implementation that is not only easy to understand but also highly performant in practice.

Simplified Parsing Expression Derivatives

“…Brzozowski [3] introduced derivative parsing for regular expressions; Might et al [10] extended this to context-free grammars. Adams et al [1] later improved the CFG derivative of Might et al, proving the same cubic worst-case time bound as shown here for PEG derivatives. Garnock-Jones et al [6] have posted a preprint of another derivative parsing algorithm for PEGs; their approach elegantly avoids defining new parsing expressions through use of a nullability combinator to represent lookahead followers as later alternatives of an alternation expression.…”

“…Derivative parsing in general is known to have poor runtime performance [10,1], as these results also demonstrate. However, this new algorithm does provide a significant improvement on the current state of the art for parsing expression derivatives, with a 40% speedup on XML and JSON, a 50% speedup on Java, and an up to 13% decrease in memory usage.…”

Derivatives of Parsing Expression Grammars

“…The most significant optimization not covered in the theoretical description of the algorithm is that the implementation takes advantage of the associativity of alternation to flatten binary trees of alternation nodes to lists of choices; this resulted in modest speedup for grammars con-taining alternations with many options. As done by Adams et al [1], the implementation also uses a memoization field in each node instead of a separate table. Limiting the number of lookahead expressions by replacing them with negative character classes, end-of-input, and until expressions was also a useful optimization, saving 20% runtime in some tests.…”

“…Might et al [13] extended the Brzozowski regular expression derivative [4] to context-free grammars; Adams et al [1] further optimized this approach and proved a cubic worst-case time bound. The derivative parsing algorithm presented here is based on their work, with extensions to account for the lookahead and ordered choice features of PEGs.…”

Staged selective parser combinators

“…Work done on parsing with derivatives [Adams et al 2016;Brzozowski 1964;Henriksen et al 2019] also has subtle connections to our work, where the CPS translation of AST to Machine is similar to differentiating with respect to that non-terminal. In particular, some derivatives work [Adams et al 2016] also uses analysis to identify the recursion points in parsers. Recently, it has been suggested that derivative-based parsing may form new foundations for parsing in general [Henriksen et al 2019]: this may suggest that CPS based abstract parsing machines are also foundational.…”