CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks

Abstract

How can deep neural networks encode information that corresponds to words in human speech into raw acoustic data? This paper proposes two neural network architectures for modeling unsupervised lexical learning from raw acoustic inputs, ciwGAN (Categorical InfoWaveGAN) and fiwGAN (Featural InfoWaveGAN), that combine a Deep Convolutional GAN architecture for audio data (WaveGAN; Donahue et al. 2019) with an information theoretic extension of GAN-InfoGAN (Chen et al., 2016), and propose a new latent space structure that can model featural learning simultaneously with a higher level classification. In addition to the Generator and the Discriminator networks, the architectures introduce a network that learns to retrieve latent codes from generated audio outputs. Lexical learning is thus modeled as emergent from an architecture that forces a deep neural network to output data such that unique information is retrievable from its acoustic outputs. The networks trained on lexical items from TIMIT learn to encode unique information corresponding to lexical items in the form of categorical variables in their latent space. By manipulating these variables, the network outputs specific lexical items. The network occasionally outputs innovative lexical items that violate training data, but are linguistically interpretable and highly informative for cognitive modeling and neural network interpretability. Innovative outputs suggest that phonetic and phonological representations learned by the network can be productively recombined and directly paralleled to productivity in human speech: a fiwGAN network trained on suit and dark outputs innovative start, even though it never saw start or even a [st] sequence in the training data. We also argue that setting latent featural codes to values well beyond training range results in almost categorical generation of prototypical lexical items and reveals underlying values of each latent code. Probing deep neural networks trained on well understood dependencies in speech bear implications for latent space interpretability, understanding how deep neural networks learn meaningful representations, as well as a potential for unsupervised text-to-speech generation in the GAN framework.

Comprehension and production of Kinyarwanda verbs in the Discriminative Lexicon

“…An unresolved question is whether this is enough. Speech recognition has been modeled with more than two layers (Beguš, Gašper 2021) with only modest success, whereas models with only two dense layers are more successful (Arnold et al 2017;Shafaei-Bajestan et al 2020). Multilingual acquisition has also sucessfully been modeled with only two layers (Chuang et al 2021), as well as several other lexical processing phenomena (Baayen et al 2019b;Baayen & Smolka 2020;Chuang et al 2020c;Tomaschek et al 2019).…”

Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication

“…In the first experiment, we use the ciwGAN (Categorical InfoWaveGAN) model proposed in Beguš (2021a). The ciwGAN model combines the WaveGAN and InfoGAN architectures.…”

“…The Discriminator/Q-network learns to retrieve the Generator's latent categorical or continuous codes (Chen et al, 2016) in addition to estimating realness of generated outputs and real training data. Beguš (2021a) proposes a model that combines these two proposals and introduces a new latent space structure (in the fiwGAN architecture). Because we are primarily interested in simple binary classification between bare and reduplicated forms, we use the ciwGAN variant of the proposal.…”

“…Recent developments in zero-resource speech modeling (Dunbar et al, 2017(Dunbar et al, , 2019(Dunbar et al, , 2020 enable modeling of speech processes in an unsupervised manner from raw acoustic data. Several proposals exist for modeling unsupervised lexical learning (Kamper et al, 2014;Lee et al, 2015;Chung et al, 2016) that include generative models such as variational autoencoders (Chung et al, 2016;Baevski et al, 2020;Niekerk et al, 2020) and GANs (Beguš, 2021a). This framework allows not only unsupervised lexical term discovery, but also phone-level identification (Eloff et al, 2019;Shain and Elsner, 2019;Chung et al, 2016;Chorowski et al, 2019).…”

“…

This paper models unsupervised learning of an identity-based pattern (or copying) in speech called reduplication from raw continuous data with deep convolutional neural networks. We use the ciwGAN architecture (Beguš, 2021a) in which learning of meaningful representations in speech emerges from a requirement that the CNNs generate informative data. We propose a technique to wug-test CNNs trained on speech and, based on four generative tests, argue that the network learns to represent an identity-based pattern in its latent space.

…”