Packrat Parsing with Elastic Sliding Window

Abstract

Packrat parsing is a linear-time implementation method of recursive descent parsers. The trick is a memoization mechanism, where all parsing results are memorized to avoid redundant parsing in cases of backtracking. An arising problem is extremely huge heap consumption in memoization, resulting in the fact that the cost of memoization is likely to outweigh its benefits. In many cases, developers need to make a difficult choice to abandon packrat parsing despite the possible exponential time parsing. Elastic packrat parsing is developed in order to avoid such a difficult choice. The heap consumption is upper-bounded since memorized results are stored on a sliding window buffer. In addition, the buffer capacity is adjusted by tracing each of nonterminal backtracking activities at runtime. Elastic packrat parsing is implemented in a part of our Nez parser. We demonstrate that the elastic packrat parsing achieves stable and robust performance against a variety of inputs with different backtracking activities.

Fast, Flexible, and Declarative Construction of Abstract Syntax Trees with PEGs

“…In the sliding window, memoized nodes are expired if the parse moves forward in the window size. Our previous work [16] confirms both the linear time parsing and the constant memory consumption if the window size is large enough to cover the length of backtracking. The Nez parser uses the sliding window for memoization, and allows the garbage collector to collect expired nodes.…”

“…Nez is a PEG-based parser generator that has a language support for the AST operators. The Nez parser is written in Java, and integrated with enhanced packrat parsing with sliding window, presented in [16], and the transactional AST machine with synchronous memoization, described in Sections 4 and 5.…”

A Declarative Extension of Parsing Expression Grammars for Recognizing Most Programming Languages

“…The execution time is measured by System.nanoTime() in Java APIs. The Nez parser that we have tested is based on [10].…”

Packrat Parsing: A Literature Review

“…In [27] concept of elastic sliding window is used and it is based upon the observation of worst longest backtrack length. Particularly author noted that if a window in the form of small memorization table slides and covers the longest backtrack then redundant calls are avoided since the storage is sufficient enough to store all the results.…”