Analyzing analytical methods: The case of phonology in neural models of spoken language

Abstract

Given the fast development of analysis techniques for NLP and speech processing systems, few systematic studies have been conducted to compare the strengths and weaknesses of each method. As a step in this direction we study the case of representations of phonology in neural network models of spoken language. We use two commonly applied analytical techniques, diagnostic classifiers and representational similarity analysis, to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. We manipulate two factors that can affect the outcome of analysis. First, we investigate the role of learning by comparing neural activations extracted from trained versus randomly-initialized models. Second, we examine the temporal scope of the activations by probing both local activations corresponding to a few milliseconds of the speech signal, and global activations pooled over the whole utterance. We conclude that reporting analysis results with randomly initialized models is crucial, and that global-scope methods tend to yield more consistent results and we recommend their use as a complement to local-scope diagnostic methods.

Acquiring language from speech by learning to remember and predict

“…In addition to reporting raw model performance, we report performance improvements from each model relative to (1) baseline U (untrained), an architecturally matched model left at random initialization (Chrupała et al, 2020), and (2) baseline X (cross-language), the architecturally matched model trained on the opposite language. 6 These two baselines quantify different contributions of the acquisition process.…”

“…Following e.g. Shain and Elsner (2019) and Chrupała et al (2020), we do so using probing classifiers. In particular, for each layer of each model's encoder, we fit linear classifiers to (1) the phoneme labels and (2) the phonological feature labels associated with the gold phoneme segment corresponding to each phone boundary.…”

Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? - A computational investigation

“…No individual study has attempted to look at the emergence of linguistic units at phonetic, syllabic, and lexical levels in a single model or study, nor compared multiple model architectures within the same experimental context. In addition, the existing studies have rarely reported baseline measures from untrained models, making it unclear how much of the findings are actually driven by the visually-guided parameter optimization compared to the effects of non-linear network dynamics also present with randomly initialized model parameters (see also Chrupała et al, 2020). This leaves unclear questions such as: 1) Can a single neural model reflect emergence of several levels of linguistic structure at the same time, including phone(me)s, syllables, and words, both in time and selectivity?…”

“…different linguistic unit types. We deliberately focus on statistical and classifier-based measures of analysis that are suitable for basic level categorical data (phone, syllable, or word types), whereas measures such as representational similarity analysis (RSA; Kriegeskorte et al, 2008) used in some other works (e.g., Chrupała et al, 2020) are better suited for non-categorical reference data 3 .…”

“…Besides analyzing the activations of individual nodes and full layers, we use two additional measures to investigate whether the full layers or their node subsets separate between different phone types: linear separability and non-linear separability, as measured by machine learning classifiers that are trained to classify phones using the activation patterns as features (also known as diagnostic classifiers; see also Belinkov & Glass, 2020;Chrupała et al, 2020). For linear separability, we use support vector machines (SVMs) with a linear kernel.…”