Neural representations for modeling variation in speech

Quantifying Language Variation Acoustically with Few Resources

“…We compute embeddings from the hidden Transformer layers of three fine-tuned deep acoustic wav2vec 2.0 large models, and subsequently determine pronunciation differences using dynamic time warping (DTW) with these embeddings (Müller, 2007). We use fine-tuned acoustic models in this study as their hidden representations were found to show the closest match with human perceptual judgements of pronunciation variation (Bartelds et al, 2022). For the transcription-based approach, we apply a (phonetically sensitive) Levenshtein distance algorithm to the available corresponding phonetic transcriptions of the 10 words in all locations.…”

“…Recently, Bartelds et al (2022) found that representations from the hidden layers of pre-trained and fine-tuned wav2vec 2.0 (large) models are suitable to represent language variation. They showed that these representations capture linguistic information that is not represented by phonetic transcriptions, while being less sensitive to non-linguistic variation in the speech signal.…”

Leveraging pre-trained representations to improve access to untranscribed speech from endangered languages

“…For BNF, we used the BUT/Phonexia feature extractor which returns for each time frame 80 activation values from a bottleneck layer originally trained for phone classi-1 https://docs.cognitive-ml.fr/shennong/ fication on the 17 languages of the IARPA Babel dataset [21]. For w2v2 features, we adapted the feature extraction code from [17], 2 which can extract outputs from the Encoder CNN (E), the Quantiser module (Q), or any one of the 24 layers of the Transformer network (T01-T24). Thus, in total we extracted features using 28 different methods for each of the 10 datasets.…”

“…Typically, representations extracted from the final layers of a Transformer network tend to be more suited to the original training task than its middle layers, which are better suited for downstream tasks [16]. Experiments in [17] investigated which of the 24 w2v2 Transformer layers may be best suited for automatic pronunciation scoring of non-native English speech. Similar to QbE-STD, the task of pronunciation scoring is a 2-stage process: features are extracted from native and non-native speech samples of the same read text, and then a DTW-based distance is calculated (lower distance indicates closer pronunciations).…”

“…Similar to QbE-STD, the task of pronunciation scoring is a 2-stage process: features are extracted from native and non-native speech samples of the same read text, and then a DTW-based distance is calculated (lower distance indicates closer pronunciations). Results from [17] showed that distances between native and non-native speech samples had a strong negative correlation with native speaker ratings of the same samples (r = -0.85; lower distances correspond to higher ratings) when computed using representations extracted from the 10th Transformer layer. In other words, the representations of the middle layers of the w2v2 Transformer network appear to be well-suited for comparing phonetic characteristics of speech samples while being robust to recording conditions and speaker characteristics.…”