Recognition is not parsing — SPPF-style parsing from cubic recognisers

Abstract

a b s t r a c tIn their recogniser forms, the Earley and RIGLR algorithms for testing whether a string can be derived from a grammar are worst-case cubic on general context free grammars (CFG). Earley gave an outline of a method for turning his recognisers into parsers, but it turns out that this method is incorrect. Tomita's GLR parser returns a shared packed parse forest (SPPF) representation of all derivations of a given string from a given CFG but is worstcase unbounded polynomial order. The parser version of the RIGLR algorithm constructs Tomita-style SPPFs and thus is also worst-case unbounded polynomial order. We have given a modified worst-case cubic GLR algorithm, that, for any string and any CFG, returns a binarised SPPF representation of all possible derivations of a given string. In this paper we apply similar techniques to develop worst-case cubic Earley and RIGLR parsing algorithms.Since Knuth's seminal 1960's work on LR parsing [16] was extended to LALR parsers by DeRemer [6,5], the Computer Science community has been able to automatically generate parsers for a very wide class of context free languages. However, many parsers are still written manually, either using tool support or even completely by hand. This is partly because in some application areas such as natural language processing and bioinformatics we do not have the luxury of designing the language so that it is amenable to known parsing techniques, but also it is clear that left to themselves computer language designers do not naturally write LR(1) grammars. Indeed, designers sometimes prefer to write ambiguous grammars because they feel that they are easier to read.A grammar not only defines the syntax of a language, it is also the starting point for the definition of the semantics, and the grammar which facilitates semantic definition is not usually the one which is LR(1). This is illustrated by the development of the Java Standard. The first edition of the Java Language Specification [8] contains a detailed discussion of the need to modify the grammar used to define the syntax and semantics in the main part of the standard to make it LALR(1) for compiler generation purposes. In the third version of the standard [9] the compiler version of the grammar is written in EBNF and is (unnecessarily) ambiguous, illustrating the difficulty of making correct transformations. Given the difficulty in constructing natural LR(1) grammars that support desired semantics, the general parsing techniques, such as the CYK [25], Earley [7] and GLR [24] algorithms, developed for natural language processing are also of interest to the wider computer science community.When using grammars as the starting point for semantics definition, we distinguish between recognisers which simply determine whether or not a given string is in the language defined by a given grammar, and parsers which also return some form of derivation of the string, if one exists. In their basic forms the CYK-, Earley-and GLR-inspired RIGLR [21] algorithms are recognisers, while standard G...

Purely functional GLL parsing

“…(A nonterminal X is cyclic in Γ if X → + Γ X.) Several technologies for representing multiple derivations efficiently are discussed in [6,12]. This paper adopts the binary subtree representation approach of [12] in which a set Υ efficiently embeds (potentially infinitely many) derivations.…”

“…Parser generators such as Yacc or Happy implement (variants of) BNF and generate parsers for different classes of contextfree grammars. Generalised parsing algorithms such as Earley [1], GLR [2] and GLL [3,4] admit all contextfree grammars and compute all possible derivations of a string [5,6]. Generalised parsing eases the design of programming languages because, compared to traditional techniques, one no longer needs to adjust the concrete syntax of the language to allow a certain parsing technique.…”

AWide-Coverage Context-Free Grammar for Icelandic and an Accompanying Parsing System

“…Tomita's parser was a Generalized LR parser of worst-case unbounded polynomial order. Later, Scott (2008) and Scott and Johnstone (2010) presented an Earley-based parser which produces a binarized SPPF representation of all derivations of a sentence in worst-case cubic time. This parser meets all of the above mentioned criteria.…”

GLL parsing with flexible combinators

“…State-of-the-art complete parsers compute Binarised Shared Packed Parse Forest (BSPPFs), a data structure capable of representing all possible derivations of a sentence in O (n 3 ) space 1 [30,33]. Rather than building BSPPFs as sets of nodes and edges, the parsing procedures of this paper construct sets of extended packed nodes 2 (EPNs) containing sufficient information to construct all other nodes and the edges between them.…”

“…Parser generators such as Yacc or Happy implement (variants of) BNF and, depending on the underlying parsing technology, generate parsers for different classes of context-free grammars. Generalised parsing algorithms such as GLR [39] and GLL [29,31] admit all context-free grammars and compute all possible derivations of an input sentence [30,33]. Several parser generators generate generalised parsers [19,24,41].…”