Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

Liu, Oli Danyi; Tang, Hao; Goldwater, Sharon

doi:10.21437/interspeech.2023-871

Cited by 3 publications

(1 citation statement)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The research community is actively working on finding ways to disentangle the representational subspaces encoding the phonetic and speaker information, e.g., Liu et al (2023). However, much less attention has been dedicated to disentangling the information pertaining to the acoustic conditions (e.g., background noise and reverberation).…”

Section: Discussionmentioning

confidence: 99%

Artificial neural networks to analyze and simulate language acquisition in children

Lavechin

2023

Preprint

View full text Add to dashboard Cite

Lightweight child-worn recorders that collect audio across an entire day allow for a big-data approach to the study of language development. By collecting the child's production and linguistic environment, these recordings offer us a uniquely naturalistic view of everyday language uses. However, such recordings quickly accumulate thousands of hours of audio and require the use of automatic speech processing algorithms. Besides providing ecologically-valid measures of what children hear and say, these recordings can fuel computational models of early language acquisition with what infants truly hear. This opens up new opportunities for running realistic language learning simulations.A first aspect of my doctoral work is dedicated to developing automatic speech processing algorithms for child-centered long-form recordings. In this manuscript, I first show that current state-of-the-art automatic speech recognition systems fail to capture the complexity of naturalistic speech as found in long-forms. I then present our attempt to propose a free, open-source, and more accurate alternative to the LENA proprietary software, which is currently the standard tool for obtaining automatic analyses of long-forms. Using supervised learning methods, my collaborators and I built a suite of speech processing tools to detect voice activity, identify voice signal sources (child vocalizations, female or male speech), count the number of linguistic units (phonemes, syllables, or words), and estimate the quantity of background noise and reverberation. A second aspect of my doctoral work is dedicated to computational models of early language acquisition. I present a first modeling study showing that self-supervised learning algorithms trained on audiobooks can learn phonetic and lexical aspects of their training language. I then show that the same algorithm trained on ecological long-forms needs inductive biases to learn phonetic aspects of its training language reliably and reflect on whether similar inductive biases may guide language learning in infants. Interestingly, there is no evidence for lexical learning on long-forms, contrary to what has been shown in the literature on more curated data. This series of studies illustrates the importance of considering ecologically-valid input data when modeling language acquisition.

show abstract

Section: Discussionmentioning

confidence: 99%

Artificial neural networks to analyze and simulate language acquisition in children

Lavechin

2023

Preprint

View full text Add to dashboard Cite

show abstract

Towards Matching Phones and Speech Representations

Yang,

Tang

2023

2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

What Do Self-Supervised Speech Models Know About Words?

Pasad,

Chien,

Settle

et al. 2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. However, these empirical successes alone do not give a complete picture of what is learned during pre-training. Recent work has begun analyzing how S3Ms encode certain properties, such as phonetic and speaker information, but we still lack a proper understanding of knowledge encoded at the word level and beyond. In this work, we use lightweight analysis methods to study segment-level linguistic properties—word identity, boundaries, pronunciation, syntactic features, and semantic features—encoded in S3Ms. We present a comparative study of layer-wise representations from ten S3Ms and find that (i) the frame-level representations within each word segment are not all equally informative, and (ii) the pre-training objective and model size heavily influence the accessibility and distribution of linguistic information across layers. We also find that on several tasks—word discrimination, word segmentation, and semantic sentence similarity—S3Ms trained with visual grounding outperform their speech-only counterparts. Finally, our task-based analyses demonstrate improved performance on word segmentation and acoustic word discrimination while using simpler methods than prior work.1

show abstract

Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

Cited by 3 publications

References 18 publications

Artificial neural networks to analyze and simulate language acquisition in children

Artificial neural networks to analyze and simulate language acquisition in children

Towards Matching Phones and Speech Representations

What Do Self-Supervised Speech Models Know About Words?

Contact Info

Product

Resources

About