Generative Adversarial Phonology: Modeling Unsupervised Phonetic and Phonological Learning With Neural Networks

Abstract

Training deep neural networks on well-understood dependencies in speech data can provide new insights into how they learn internal representations. This paper argues that acquisition of speech can be modeled as a dependency between random space and generated speech data in the Generative Adversarial Network architecture and proposes a methodology to uncover the network's internal representations that correspond to phonetic and phonological properties. The Generative Adversarial architecture is uniquely appropriate for modeling phonetic and phonological learning because the network is trained on unannotated raw acoustic data and learning is unsupervised without any language-specific assumptions or pre-assumed levels of abstraction. A Generative Adversarial Network was trained on an allophonic distribution in English, in which voiceless stops surface as aspirated word-initially before stressed vowels, except if preceded by a sibilant [s]. The network successfully learns the allophonic alternation: the network's generated speech signal contains the conditional distribution of aspiration duration. The paper proposes a technique for establishing the network's internal representations that identifies latent variables that correspond to, for example, presence of [s] and its spectral properties. By manipulating these variables, we actively control the presence of [s] and its frication amplitude in the generated outputs. This suggests that the network learns to use latent variables as an approximation of phonetic and phonological representations. Crucially, we observe that the dependencies learned in training extend beyond the training interval, which allows for additional exploration of learning representations. The paper also discusses how the network's architecture and innovative outputs resemble and differ from linguistic behavior in language acquisition, speech disorders, and speech errors, and how well-understood dependencies in speech data can help us interpret how neural networks learn their representations.

CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks

“…The result of the training in the architecture outlined in Figure 1 is a Generator network that outputs raw acoustic data that resemble real data from the TIMIT database, such that the Discriminator becomes unsuccessful in assigning "realness" scores (Brownlee, 2019). Crucially, unlike in other architectures, the Generator's outputs are never a full replication of the input: the Generator outputs innovative data that resemble input data, but also violate many of the distributions in a linguistically interpretable manner (Beguš, 2020). In addition to outputting innovative data that resemble speech in the input, the Generator also learns to associate each lexical item with a unique code in its latent space.…”

“…Language acquisition has, to the author's knowledge, not been modeled with the GAN architecture prior to Beguš (2020), despite several aspects of the architecture that can be paralleled to language acquisition. Beguš (2020) proposes that phonetic and phonological learning can simultaneously be modeled as a dependency between latent space and output data in Deep Convolutional Generative Adversarial Networks (Goodfellow et al, 2014;Radford et al, 2015;Donahue et al, 2019). Unlike in the autoencoder architectures, the outputs of the GAN models are innovative, not directly connected to the inputs, and violate training data distributions in highly informative ways.…”

“…Despite their several advantages, to our knowledge, lexical learning has not yet been modeled with unsupervised generative deep convolutional neural network models. In this paper, we follow the proposal in Beguš (2020) that phonetic and phonological acquisition can be modeled as a dependency between latent space and generated data in the GAN architecture and add lexical learning component to the model. We modify the WaveGAN architecture and add the InfoGAN's Q-network (based partially on implementation in Rodionov 2018) to computationally simulate lexical learning from raw acoustic data.…”

“…One of the advantages of the proposal in Beguš (2020) is that the model learns phonological alternations, i.e. context-dependent changes in realization of speech sounds, simultaneously with learning of acoustic properties of human speech.…”

“…The latent variables that correspond to features can be actively manipulated to generate data with or without some phonetic/phonological properties. These representations, however, are limited to the phonetic/phonological level exclusively in Beguš (2020) and contain no lexical information.…”