Strengths and weaknesses of finite-state technology: a case study in morphological grammar development

Abstract

Finite-state technology is considered the preferred model for representing the phonology and morphology of natural languages. The attractiveness of this technology for natural language processing stems from four sources: modularity of the design, due to the closure properties of regular languages and relations; the compact representation that is achieved through minimization; efficiency, which is a result of linear recognition time with finite-state devices; and reversibility, resulting from the declarative nature of such devices. However, when widecoverage morphological grammars are considered, finite-state technology does not scale up well, and the benefits of this technology can be overshadowed by the limitations it imposes as a programming environment for language processing. This paper investigates the strengths and weaknesses of existing technology, focusing on various aspects of large-scale grammar development. Using a real-world case study, we compare a finite-state implementation with an equivalent Java program with respect to ease of development, modularity, maintainability of the code, and space and time efficiency. We identify two main problems, abstraction and incremental development, which are currently not addressed sufficiently well by finite-state technology, and which we believe should be the focus of future research and development.

Arabic spelling error detection and correction

“…AraComLex contains 30,587 lemmas and is developed using finite state technology. There are a number of advantages of finite-state technology that makes it especially attractive in dealing with human language morphologies (Wintner 2008). They include bidirectionality and the ability to generate as straightforwardly as to analyze.…”

A Framework for the Generation of Computer System Diagnostics in Natural Language using Finite State Methods

“…The natural language explanation is obtained by composing FS transducers in a pipeline. FS technologies are best-known for the representation of certain kinds of linguistic knowledge, most notably morphology (Wintner, 2008). In contrast, we used XFST to implement linguistic techniques such as structuring the text into paragraphs, aggregation, contextuality -as previously illustrated.…”

Morphological disambiguation of Hebrew: a case study in classifier combination

“…Our departure point in this work is the MILA morphological analyzer Itai and Wintner (2008), a wide coverage, linguistically motivated morphological analyzer of Hebrew, which is based on a finite-state morphological grammar (Yona and Wintner, 2008) but is reimplemented in Java (Wintner, 2008). The output that the analyzer produces for the form $bth is illustrated in Table 1.…”