RNGSGLR: Generalization of the Context-Aware Scanning Architecture for All Character-Level Context-Free Languages

Abstract

The limitations of traditional parsing architecture are well known. Even when paired with parsing methods that accept all context-free grammars (CFGs), the resulting combination for any given CFG accepts only a limited subset of corresponding character-level context-free languages (CFL). We present a novel scanner-based architecture that for any given CFG accepts all corresponding character-level CFLs. It can directly parse all possible specifications consisting of a grammar and regular definitions. The architecture is based on right-nulled generalized LR (RNGLR) parsing and is a generalization of the context-aware scanning architecture. Our architecture does not require any disambiguation rules to resolve lexical conflicts, it conceptually has an unbounded parser and scanner lookahead and it is streaming. The added robustness and flexibility allow for easier grammar development and modification.

On Parsing Programming Languages with Turing-Complete Parser

“…In general, the syntax of programming and most domain-specific languages is described by context-free grammars [19], while common non-context-free features of these languages, e.g., scopes and namespaces, are usually dealt with throughout a lexical, syntax and semantic analysis [18,20,21]. These features are dealt with informal techniques as standard lexers, or the context-free parser can be augmented with context-aware lexers [22][23][24].…”

“…Woods algorithm [40] CYK-like tabular algorithm [41] CS reduction systems [8] Limited context-sensitive Weakly context-sensitive languages [42] Backtracking LR [29] (can be done safely) Loop-free context-sensitive languages [43] Principled stateful parser [30] ALL( * ) [4,5] Data-dependent grammars [31,32] PEG parsing [34] Context-aware lexers [22][23][24] However, most parser algorithms used nowadays are descendants of relatively old algorithms, e.g., the canonical LR and LL parsing [44,45] or even PEG parsing [46]. However, although extensively studied [17,18], error recovery has never been formalized to the same extend as parsing.…”

Context-sensitive parsing for programming languages

“…In a compiler, these non-context-free issues are resolved during or even after lexical and syntax analysis, e.g., during semantic analysis [17][18][19]. More precisely, context-sensitive features of programming languages has been dealt with in three different ways: (a) with ad-hoc solutions built into a standard lexer or context-free parser, (b) with context-aware lexers [20][21][22], or (c) with enhanced context-free parsers capable of handling specific types of more complex language features.…”

“…and only if that move exists in . Hence, sequence of moves(22) can be extended with a move𝑞¢𝑋 1 𝑋 2 … 𝑋 𝑛 $ ⊢ ¢𝑞 ′ 𝑋 1 𝑋 2 … 𝑋 𝑛 $ and reduction process (23) can be extended with [[⟨𝑞 ′ ∶ 𝑋 1 ⟩⟨𝑋 2 ⟩ ⟨𝑋 3 ⟩ ⋯ ⟨𝑋 𝑛 ⟩]].Thus the extended sequence of moves matches reduction process(24) for 𝑖 = 1.• For 𝑘 + 1 and 1 ≤ 𝑖 ≤ 𝑛 the head can be moved either right or left but there can be no move available if the input string𝑎 1 𝑎 2 … 𝑎 𝑛 ∉ 𝐿(). Based on (18) there is 𝛿(𝑞, 𝑋 𝑖 ) = ⟨𝑞 ′ , 𝑋 ′ 𝑖 , R⟩ in  if and only if 𝑅 contains either ⟨𝑞 ∶ 𝑋 𝑖 ⟩⟨𝑋 𝑖+1 ⟩ ↦ ⟨𝑋 ′ 𝑖 ⟩⟨𝑞 ′ ∶ 𝑋 𝑖+1 ⟩ if 𝑖 < 𝑛 or ⟨𝑞 ∶ 𝑋 𝑖 ⟩]] ↦ ⟨𝑞 ⊳ 𝑋 ′ 𝑖 ⟩]] if 𝑖 = 𝑛.Hence, sequence of moves (22) can be extended with a move𝑋 0 𝑋 1 … 𝑋 𝑖−1 𝑞𝑋 𝑖 𝑋 𝑖+1 … 𝑋 𝑛+1 ⊢ 𝑋 0 𝑋 1 … 𝑋 𝑖−1 𝑋 ′ 𝑖 𝑞 ′ 𝑋 𝑖+1 … 𝑋 𝑛+1and a reduction process (24) can be extended with[[⟨𝑋 1 ⟩⟨𝑋 2 ⟩ ⋯ ⟨𝑋 𝑖−1 ⟩⟨𝑋 ′ 𝑖 ⟩⟨𝑞 ′ ∶ 𝑋 𝑖+1 ⟩ ⋯ ⟨𝑋 𝑛 ⟩]] if 𝑖 < 𝑛 or [[⟨𝑋 1 ⟩⟨𝑋 2 ⟩ ⋯ ⟨𝑋 𝑛−1 ⟩⟨𝑞 ′ ⊳ 𝑋 ′ 𝑛 ⟩]] if 𝑖 = 𝑛.Thus the extended sequence of moves matches reduction process(24) if 𝑖 < 𝑛 or (24) if 𝑖 = 𝑛.…”