A Modified Earley Parser for Huge Natural Language Grammars

Abstract

For almost a half century Earley parser has been used in the parsing of context-free grammars and it is considered as a touch-stone algorithm in the history of parsing algorithms. On the other hand, it is also known for being expensive from its time requirement and memory usage perspectives. For huge context-free grammars, its performance is not good since its time complexity also depends on the number of rules in the grammar. The time complexity of the original Earley parser is O(R 2 N 3 ) where N is the string length, and R is the number of rules. In this paper, we aim to improve time and memory usage performances of Earley parser for grammars with a large number of rules. In our approach, we prefer radix tree representation for rules instead of list representation as in original Earley parser. We have tested our algorithm using different number of rule sets up to 200,000 which are all learned by an examplebased machine translation system. According to our evaluation results, our modified parser has a time bound of O(log(R)N 3 ), and it has 20% less memory usage regarding the original Earley parser.

Bidirectional Machine Translation Between Turkish and Turkish Sign Language : A Data-Driven Approach

“…Though our purpose might seem irrelevant at first sight, in this study we use a grammar parser to find possible translations. We modified the original Earley parser to make it much faster in matching and more efficient in memory usage for our purpose [30].…”

“…After an input sentence passes from the LSC, a radix tree carrying all relevant grammar rules is generated. Relevant rules are gathered from the generalized translation examples which carry similar parts to the input sentence [30]. Our parser, or matcher, scans relevant rules coming with a radix tree, makes use of the lattice files when type information is needed, and tries to find a candidate translation for the sentence in source language.…”

Efficient Semiring-Weighted Earley Parsing

“…A fast implementation of Earley's algorithm is reported byPolat et al (2016) but does not appear to be public.…”