Supercombinator Set Construction from a Context-Free Representation of Text

Abstract

Abstract-Grammars might be used for various other aspects, than just to represent a language. Grammar inference is a large field which main goal is the construction of grammars from various sources. Written text might be analysed indirectly with the use of such inferred grammars. Grammars acquired from processed text might grow into large structures as the inference process could be continuous. We present a method to decompose and store grammars into a non-redundant set of lambda calculus supercombinators. Grammars decomposition is based on their structure and each distinct element is stored only once in such a structure. We present a method that can create such a set from any context-free grammar. To prove this and to show the possible applications in the field of natural language processing we present a case study performed on samples from two books. Those samples are the entire Book of Genesis from The King James Bible and the first 24 chapters of War and peace by Tolstoy. We obtain context-free grammars with the Sequitur algorithm and then we process them with our method. The results show significant decline in the number of grammar elements in all cases.

Properties and Limits of Supercombinator Set Acquired from Context-free Grammar Samples

“…As already mentioned, we are no longer using grammars generated with the Sequitur algorithm as we have done in [4]. We have decided to use different kind of input data.…”

“…In Table. III 4 we see the amount of supercombinators taken from our set split by arity up to the number of 16. We obtain all theoretically possible supercombinators up to the arity of 6.…”

“…The results presented in this paper are achieved with the use of our algorithm that has been improved and now differs in some points from the one presented in [4]. We have unified the merge operation and also used more efficient data structures that increased the speed of grammar processing.…”

“…We have shown in our previous work [4] that any Contextfree grammar (CFG) can be transformed into a supercombinator form. Which means that we can abstract the structure from the data (represented in grammars as terminal symbols).…”

“…• We present updated algorithm for supercombinator form acquisition that runs more smoothly than the one from our previous work [4]. Short description of its basic functionality and a list of performed changes are shown in the section III.…”