One parser to rule them all

Abstract

Despite the long history of research in parsing, constructing parsers for real programming languages remains a difficult and painful task. In the last decades, different parser generators emerged to allow the construction of parsers from a BNF-like specification. However, still today, many parsers are handwritten, or are only partly generated, and include various hacks to deal with different peculiarities in programming languages. The main problem is that current declarative syntax definition techniques are based on pure contextfree grammars, while many constructs found in programming languages require context information.In this paper we propose a parsing framework that embraces context information in its core. Our framework is based on data-dependent grammars, which extend contextfree grammars with arbitrary computation, variable binding and constraints. We present an implementation of our framework on top of the Generalized LL (GLL) parsing algorithm, and show how common idioms in syntax of programming languages such as (1) lexical disambiguation filters, (2) operator precedence, (3) indentation-sensitive rules, and (4) conditional preprocessor directives can be mapped to datadependent grammars. We demonstrate the initial experience with our framework, by parsing more than 20 000 Java, C#, Haskell, and OCaml source files.

Practical, general parser combinators

“…Among all the related work we discussed so far, GLL parsing is the closest to our work, especially if we consider a version of GLL [2] which uses a more efficient GSS [1]. In fact, CPS parsers have the same performance characteristics as GLL parsers with the new GSS [1].…”

“…Such interpretive version of Earley parsing is provided in [20]. In an earlier work [2], we provided an interpretive formulation of GLL Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.…”

“…However, such parser combinators still have the flavor of a parser generator. Moreover, an extension to such parser combinators may also require modification to the underlying parsing algorithm, for example, the modified Earley sets [11] and modified GLL algorithm [2] to support data dependency. In contrast, parser combinators that provide shallow embedding enable directly executable parsers, as grammars are represented directly as functions.…”

Nez: practical open grammar language

“…• Python3 -Ported from Python 3.0 abstract grammar 3 . The Indent is used to express the indentation-based code block by capturing white spaces.…”

“…Data-dependent grammars [3,13] share similar ideas in terms of recognizing context-sensitive syntax. Briefly, these grammars uses a variable to be bound to a parsed result, which seems to be the addition of symbols in Nez.…”