Representational bias in unsupervised learning of syllable structure

Abstract

Unsupervised learning algorithms based on Expectation Maximization (EM) are often straightforward to implement and provably converge on a local likelihood maximum. However, these algorithms often do not perform well in practice. Common wisdom holds that they yield poor results because they are overly sensitive to initial parameter values and easily get stuck in local (but not global) maxima. We present a series of experiments indicating that for the task of learning syllable structure, the initial parameter weights are not crucial. Rather, it is the choice of model class itself that makes the difference between successful and unsuccessful learning. We use a language-universal rule-based algorithm to find a good set of parameters, and then train the parameter weights using EM. We achieve word accuracy of 95.9% on German and 97.1% on English, as compared to 97.4% and 98.1% respectively for supervised training.

On the syllabification of phonemes

“…Applying a standard smoothing algorithm and fourth-order HMM, Demberg (2006) scores 98.47% word accuracy. A fifth-order joint N -gram model of Schmid et al (2007) Table 4: Word accuracy on the datasets of Goldwater and Johnson (2005).…”

“…The results are somewhat lower than in (Müller, 2001), but the approach can be more easily ported across languages. Goldwater and Johnson (2005) also explore using EM to learn the structure of English and German phonemes in an unsupervised setting, following Müller in modeling syllable structure with PCFGs. They initialize their parameters using a deterministic parser implementing the sonority principle and estimate the parameters for their maximum likelihood approach using EM.…”

“…The numbers that follow refer to 10-fold cross-validation on the entire lexicon (over 320K entries) unless noted otherwise. Krenn (1997) Goldwater and Johnson (2005).…”

A procedure for estimating gestural scores from speech acoustics

“…3. First, the ARPABET transcription of a phone structure {A, w} for each word is explicitly parsed into syllables by means of a syllabification algorithm based on the English phonotactics principle of the maximization of syllable onsets (Goldwater and Johnson, 2005). Then, from the syllabified ARPABET transcription, the linguistic gestural model generates a synthesized gestural score, fĜ;Bg, specifying the CL and CD targets and stiffnesses for each TV, and the times during which those targets are in force.…”

Dual stream speech recognition using articulatory syllable models

“…For training data we use the BN-VL task of N-Best phonetically segmented with HMM models. For segmenting the vocabulary we use an unsupervised syllabification (Goldwater and Johnson 2005) method we are publishing in a separate paper. The method syllabifies Dutch word lists with a 90% word accuracy with the bulk of errors being in one-phoneme shifts in syllable boundaries of the rarer long words.…”