Autoregressive Predictive Coding: A Comprehensive Study

Yang, Gene-Ping; Yeh, Sung-Lin; Chung, Yu-An; Glass, James; Tang, Hao

doi:10.1109/jstsp.2022.3203608

Cited by 12 publications

(4 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both CPC-big and CPC-small are taken from the Zero Speech 2021 baseline 3 . We use our own implementation of APC, following [15]. The CPC and APC models differ in several ways.…”

Section: Methodsmentioning

confidence: 99%

“…We follow recent work in speech technology and machine learning in using the term predictive coding to refer to error-driven learning based on forward prediction, contrasting with masked prediction or other objectives. In the more general sense used in information theory[14,15] and computational neuroscience[16,17,18], masked prediction can also be viewed as a type of predictive coding.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

Liu,

Tang,

Goldwater

2023

Interspeech 2023

View full text Add to dashboard Cite

Self-supervised speech representations are known to encode both speaker and phonetic information, but how they are distributed in the high-dimensional space remains largely unexplored. We hypothesize that they are encoded in orthogonal subspaces, a property that lends itself to simple disentanglement. Applying principal component analysis to representations of two predictive coding models, we identify two subspaces that capture speaker and phonetic variances, and confirm that they are nearly orthogonal. Based on this property, we propose a new speaker normalization method which collapses the subspace that encodes speaker information, without requiring transcriptions. Probing experiments show that our method effectively eliminates speaker information and outperforms a previous baseline in phone discrimination tasks. Moreover, the approach generalizes and can be used to remove information of unseen speakers.

show abstract

“…Both CPC-big and CPC-small are taken from the Zero Speech 2021 baseline 3 . We use our own implementation of APC, following [15]. The CPC and APC models differ in several ways.…”

Section: Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

Liu,

Tang,

Goldwater

2023

Interspeech 2023

View full text Add to dashboard Cite

show abstract

“…Following [14,6], we fix the time shift k to 5, choosing the recurrent network as 3-layer 512-dimensional unidirectional LSTMs. We will compare training the model with VQ-APC and neural HMMs.…”

Section: Methodsmentioning

confidence: 99%

“…where W is a linear projection, V is the codebook, and d is the dimension of x t . We simply choose W the identity matrix, setting the codeword dimension to d. The choice of Gaussian is aligned with the L 2 loss in [5,14]. Note that the parametrization does not introduce any additional parameters compared to VQ-APC.…”

Section: Neural Parametrizationmentioning

confidence: 99%

Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models

Yeh¹,

Tang²

2022

Preprint

View full text Add to dashboard Cite

While discrete latent variable models have had great success in self-supervised learning, most models assume that frames are independent. Due to the segmental nature of phonemes in speech perception, modeling dependencies among latent variables at the frame level can potentially improve the learned representations on phonetic-related tasks. In this work, we assume Markovian dependencies among latent variables, and propose to learn speech representations with neural hidden Markov models. Our general framework allows us to compare to self-supervised models that assume independence, while keeping the number of parameters fixed. The added dependencies improve the accessibility of phonetic information, phonetic segmentation, and the cluster purity of phones, showcasing the benefit of the assumed dependencies.

show abstract

Simulating prenatal language exposure in computational models: An exploration study

Cruz Blandón,

Gonzalez-Gomez,

Lavechin

et al. 2025

Cognition

View full text Add to dashboard Cite

Autoregressive Predictive Coding: A Comprehensive Study

Cited by 12 publications

References 51 publications

Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces

Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models

Simulating prenatal language exposure in computational models: An exploration study

Contact Info

Product

Resources

About