Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation

Kreuk, Felix; Keshet, Joseph; Adi, Yossi

doi:10.48550/arxiv.2007.13465

Cited by 6 publications

(29 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The frames with high dissimilarity are considered segment boundary candidates. Similar to [23], [35], [36], we apply a peak detection algorithm to find the final segment boundaries. The peak prominence value for the peak detection algorithm is fined-tuned on the validation dataset [23].…”

Section: E Inferencementioning

confidence: 99%

“…Similar to [23], [35], [36], we apply a peak detection algorithm to find the final segment boundaries. The peak prominence value for the peak detection algorithm is fined-tuned on the validation dataset [23]. For word segmentation, the model outputs a dissimilarity score between context representation and segment representation.…”

Section: E Inferencementioning

confidence: 99%

“…For the TIMIT dataset, the standard train/test split was used, with 10% of the train data randomly sampled employed as the validation subset. We split the data into 80/10/10 for train, test, and validation sets for Buckeye, similarly to [23]. The longer recordings were divided into smaller ones by cutting during untranscribed fragments, noises, and silences to ease training.…”

Section: Experiments a Datasets And Evaluation Metricsmentioning

confidence: 99%

“…SSL has been shown to be effective for learning representations in natural language processing [24], [25], images [26], and speech processing [21], [22]. While most SSL work in speech focuses on learning representations [1], [21], [22], SSL has recently been used to identify the spectral changes from the raw waveform and detect phone boundaries [23].…”

Section: Introductionmentioning

confidence: 99%

“…The SSL framework exploits the temporal structure present in the speech to learn latent representations. For instance, in [23], the auxiliary task is to identify the next frame's latent representation given the latent representation of a reference frame. A feature extractor, e.g., Convolutional Neural Network (CNN), that maps the speech signal to a latent space is optimized using the Noise Contrastive Estimation (NCE) [27] to correctly identify the next frame within a set of frames that includes random distractor frames.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Coding

Bhati¹,

Villalba²,

Żelasko³

et al. 2021

Preprint

View full text Add to dashboard Cite

Typically, unsupervised segmentation of speech into the phone and word-like units are treated as separate tasks and are often done via different methods which do not fully leverage the inter-dependence of the two tasks. Here, we unify them and propose a technique that can jointly perform both, showing that these two tasks indeed benefit from each other. Recent attempts employ self-supervised learning, such as contrastive predictive coding (CPC), where the next frame is predicted given past context. However, CPC only looks at the audio signal's frame-level structure. We overcome this limitation with a segmental contrastive predictive coding (SCPC) framework to model the signal structure at a higher level, e.g., phone level. A convolutional neural network learns frame-level representation from the raw waveform via noise-contrastive estimation (NCE). A differentiable boundary detector finds variable-length segments, which are then used to optimize a segment encoder via NCE to learn segment representations. The differentiable boundary detector allows us to train frame-level and segment-level encoders jointly. Experiments show that our single model outperforms existing phone and word segmentation methods on TIMIT and Buckeye datasets. We discover that phone class impacts the boundary detection performance, and the boundaries between successive vowels or semivowels are the most difficult to identify. Finally, we use SCPC to extract speech features at the segment level rather than at uniformly spaced frame level (e.g., 10 ms) and produce variable rate representations that change according to the contents of the utterance. We can lower the feature extraction rate from the typical 100 Hz to as low as 14.5 Hz on average while still outperforming the MFCC features on the linear phone classification task.

show abstract

Section: E Inferencementioning

confidence: 99%

Section: E Inferencementioning

confidence: 99%

Section: Experiments a Datasets And Evaluation Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Coding

Bhati¹,

Villalba²,

Żelasko³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Generative Spoken Language Model based on continuous word-sized audio tokens

Algayres,

Adi,

Nguyen

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

In NLP, text language models based on words or subwords are known to outperform their character-based counterparts. Yet, in the speech community, the standard input of spoken LMs are 20ms or 40ms-long discrete units (shorter than a phoneme). Taking inspiration from wordbased LM, we introduce a Generative Spoken Language Model (GSLM) based on word-size continuous-valued audio embeddings that can generate diverse and expressive language output. This is obtained by replacing lookup table for lexical types with a Lexical Embedding function, the cross entropy loss by a contrastive loss, and multinomial sampling by k-NN sampling. The resulting model is the first generative language model based on word-size continuous embeddings. Its performance is on par with discrete unit GSLMs regarding generation quality as measured by automatic metrics and subjective human judgements. Moreover, it is five times more memory efficient thanks to its large 200ms units. In addition, the embeddings before and after the Lexical Embedder are phonetically and semantically interpretable. 1

show abstract

Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation

et al. 2021

View full text Add to dashboard Cite

Automatic detection of phoneme or word-like units is one of the core objectives in zero-resource speech processing. Recent attempts employ self-supervised training methods, such as contrastive predictive coding (CPC), where the next frame is predicted given past context. However, CPC only looks at the audio signal's frame-level structure. We overcome this limitation with a segmental contrastive predictive coding (SCPC) framework that can model the signal structure at a higher level e.g. at the phoneme level. In this framework, a convolutional neural network learns frame-level representation from the raw waveform via noise-contrastive estimation (NCE). A differentiable boundary detector finds variable-length segments, which are then used to optimize a segment encoder via NCE to learn segment representations. The differentiable boundary detector allows us to train frame-level and segment-level encoders jointly. Typically, phoneme and word segmentation are treated as separate tasks. We unify them and experimentally show that our single model outperforms existing phoneme and word segmentation methods on TIMIT and Buckeye datasets. We analyze the impact of boundary threshold and when is the right time to include the segmental loss in the learning process.

show abstract

Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation

Cited by 6 publications

References 23 publications

Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Coding

Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Coding

Generative Spoken Language Model based on continuous word-sized audio tokens

Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation

Contact Info

Product

Resources

About