Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication

Abstract

This paper models unsupervised learning of an identity-based pattern (or copying) in speech called reduplication from raw continuous data with deep convolutional neural networks. We use the ciwGAN architecture (Beguš, 2021a) in which learning of meaningful representations in speech emerges from a requirement that the CNNs generate informative data. We propose a technique to wug-test CNNs trained on speech and, based on four generative tests, argue that the network learns to represent an identity-based pattern in its latent space. By manipulating only two categorical variables in the latent space, we can actively turn an unreduplicated form into a reduplicated form with no other substantial changes to the output in the majority of cases. We also argue that the network extends the identity-based pattern to unobserved data. Exploration of how meaningful representations of identity-based patterns emerge in CNNs and how the latent space variables outside of the training range correlate with identity-based patterns in the output has general implications for neural network interpretability.

Encoding of speech in convolutional layers and the brain stem based on language experience

“…F1, perhaps also F2; Krishnan 2002), “acoustic onsets” such as burst, and “frequency transitions”(Abrams and Kraus, 2015). Similarly, it has been shown that the same acoustic properties are encoded in the second to last convolutional layer: periodicity and F0 together with F0 transitions, low frequency formant structure (F1 and to lesser degree F2), burst, and timing of individual segments (Begušs and Zhou, 2021b,a). Figures 1a,b,c,d; 3a,b,c,d; and 4a,b,c,d,e illustrate how the signal from Conv1/4 and cABR encode the same acoustic properties.…”

“…Figures 1a,b,c,d; 3a,b,c,d; and 4a,b,c,d,e illustrate how the signal from Conv1/4 and cABR encode the same acoustic properties. Higher-level convolutional layers do not encode all these acoustic properties (Begušs and Zhou, 2021b,a). In sum, both the earlier intermediate layers and cABR signal feature encoding of the same acoustic properties (F0, burst, timing, and low frequency formant structure).…”

“…The second to last convolutional layer (Conv4 in the Generator and Conv1 in the Discriminator) have 64 filters which result in 64 feature maps for each input/output. Individual feature maps offer limited interpretability, but a simple averaged sum over all feature maps after ReLU or Leaky ReLU activation offers highly interpretable time series data (Begušs and Zhou, 2021b,a).…”

“…The fourth and first convolutional layers, respectively, are analyzed according to Begušs and Zhou (2021b,a) by averaging over all feature maps after ReLU or Leaky ReLU activations. 3 This results in a time series t for each Convolutional layer as in (2) from Beguš and Zhou (2021a).…”

“…By training the networks on two languages with a different encoding of a phonetic property (as confirmed by brain experiments), we not only test the encoding of acoustics, but we test specific phonetic features that constitute phonological contrasts: the distinction between voiceless stops (such as [t]) and voiced stops (such as [d]) in English and Spanish. How phonological (meaning-distinguishing) contrasts are encoded in the brain Zhao and Kuhl (2018) and in deep neural network trained on speech Beguš (2020b, 2021b,a,c) can yield new information on encoding of linguistically meaningful units across the two systems.…”