Maximum Likelihood Estimation of Factored Regular Deterministic Stochastic Languages

Abstract

This paper proves that for every class C of stochastic languages defined with the coemission product of finitely many probabilistic, deterministic finite-state acceptors (PDFA) and for every data sequence D of finitely many strings drawn i.i.d. from some stochastic language, the Maximum Likelihood Estimate of D with respect to C can be found efficiently by locally optimizing the parameter values. We show that a consequence of the co-emission product is that each PDFA behaves like an independent factor in a joint distribution. Thus, the likelihood function decomposes in a natural way. We also show that the negative log likelihood function is convex. These results are motivated by the study of Strictly k-Piecewise (SP k) Stochastic Languages, which form a class of stochastic languages which is both linguistically motivated and naturally understood in terms of the coemission product of certain PDFAs.

Learning Nonlocal Phonotactics in a Strictly Piecewise Probabilistic Phonotactic Model

“…The learning problem for any subregular languages is the same as SP language (Θ = arg min Θ M (nll(S|Θ M ))). Moreover, Shibata & Heinz (2019) has shown the convexity of this learning problem as long as the production of word likelihood of specific subregular language is defined with respect to coemission probability. One can easily modify each factored WDFA in SP phonotactic model to represent one tier, and model multitier interactions through coemission probability.…”

Simple induction of (deterministic) probabilistic finite-state automata for phonotactics by stochastic gradient descent

“…In our framework, it is possible to induce an unrestricted automaton with a given number of states, or an automaton with hard-coded constraints representing various subregular languages. This work fills a gap in the formal linguistics literature, where learners have been developed within certain subregular classes (Shibata and Heinz, 2019;Heinz, 2010;Heinz and Rogers, 2010;Futrell et al, 2017), whereas our learner can in principle induce any (sub)regular language. In addition, we demonstrate how Strictly Local and Strictly Piecewise constraints can be encoded within our framework, and show how informationtheoretic regularization can be applied to produce deterministic automata.…”

“…Strictly Piecewise A Strictly k-Piecewise k-SP) language, each symbol depends on the presence of any preceding k − 1 symbols at arbitrary distance (Heinz, 2007(Heinz, , 2018Shibata and Heinz, 2019). For example, in a 2-SP language, in a string abc, c would be conditional on the presence of a and the presence of b, without regard to distance nor the relative order of a and b.…”

“…Induction of restricted PDFAs, especially for SL and SP languages, is explored in Rogers (2013, 2010) Our work differs from previous approaches in its simplicity. Inspired by Shibata and Heinz (2019), we optimize the training objective directly via gradient descent, without approximations or heuristics other than the use of minibatches. The same algorithm is applied to learn both transition and emission structure, for learning of both general PFAs and restricted PDFAs.…”

Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

“…Strictly Piecewise A Strictly k-Piecewise k-SP) language, each symbol depends on the presence of any preceding k − 1 symbols at arbitrary distance Shibata and Heinz, 2019). For example, in a 2-SP language, in a string abc, c would be conditional on the presence of a and the presence of b, without regard to distance nor the relative order of a and b.…”