Estimating articulatory movements in speech production with transformer networks

Udupa, Sathvik; Roy, Anwesha; Singh, Abhayjeet; Illa, Aravind; Ghosh, Prasanta Kumar

doi:10.48550/arxiv.2104.05017

“…1, we use a transformer architecture to represent the input phoneme sequence. Transformers are an ideal choice since they have been shown to perform well in learning text features, especially in multi-modal scenarios [15,16]. We then use a custom convolutional decoder consisting of 2D and 3D convolutional neural networks (CNN) to represent the frames.…”

Section: Proposed Methodologymentioning

confidence: 99%

“…Deep learning methods are appropriate for this task since the models can learn across long sequences of arbitrarily different lengths, and can learn between modalities and generalise well to unseen situations. In recent years, transformer neural networks have been shown to perform well on various speech-based tasks involving sequence-to-sequence modelling such as speech synthesis [15], acoustic to articulatory inversion [16] etc. Self-attention operation in transformer networks enables learning the dependencies between positions in a sequence, making it suitable for learning good phoneme-level features.…”

Section: Introductionmentioning

confidence: 99%

Real-Time MRI Video Synthesis from Time Aligned Phonemes with Sequence-to-Sequence Networks

Udupa

¹

,

Ghosh

²

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

1

0

View full text Add to dashboard Cite

Real-Time Magnetic resonance imaging (rtMRI) of the midsagittal plane of the mouth is of interest for speech production research. In this work, we focus on estimating utterance level rtMRI video from the spoken phoneme sequence. We obtain time-aligned phonemes from forced alignment, to obtain frame-level phoneme sequences which are aligned with rtMRI frames. We propose a sequence-tosequence learning model with a transformer phoneme encoder and convolutional frame decoder. We then modify the learning by using intermediary features obtained from sampling from a pretrained phoneme-conditioned variational autoencoder (CVAE). We train on 8 subjects in a subject-specific manner and demonstrate the performance with a subjective test. We also use an auxiliary task of air tissue boundary (ATB) segmentation to obtain the objective scores on the proposed models. We show that the proposed method is able to generate realistic rtMRI video for unseen utterances, and adding CVAE is beneficial for learning the sequence-to-sequence mapping for subjects where the mapping is hard to learn.

show abstract

“…1, we use a transformer architecture to represent the input phoneme sequence. Transformers are an ideal choice since they have been shown to perform well in learning text features, especially in multi-modal scenarios [15,16]. We then use a custom convolutional decoder consisting of 2D and 3D convolutional neural networks (CNN) to represent the frames.…”

Section: Proposed Methodologymentioning

confidence: 99%

“…Deep learning methods are appropriate for this task since the models can learn across long sequences of arbitrarily different lengths, and can learn between modalities and generalise well to unseen situations. In recent years, transformer neural networks have been shown to perform well on various speech-based tasks involving sequence-to-sequence modelling such as speech synthesis [15], acoustic to articulatory inversion [16] etc. Self-attention operation in transformer networks enables learning the dependencies between positions in a sequence, making it suitable for learning good phoneme-level features.…”

Section: Introductionmentioning

confidence: 99%

Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networks

Udupa¹,

Ghosh²

2022

Preprint

0

View full text Add to dashboard Cite

Real-Time Magnetic resonance imaging (rtMRI) of the midsagittal plane of the mouth is of interest for speech production research. In this work, we focus on estimating utterance level rtMRI video from the spoken phoneme sequence. We obtain time-aligned phonemes from forced alignment, to obtain frame-level phoneme sequences which are aligned with rtMRI frames. We propose a sequence-tosequence learning model with a transformer phoneme encoder and convolutional frame decoder. We then modify the learning by using intermediary features obtained from sampling from a pretrained phoneme-conditioned variational autoencoder (CVAE). We train on 8 subjects in a subject-specific manner and demonstrate the performance with a subjective test. We also use an auxiliary task of air tissue boundary (ATB) segmentation to obtain the objective scores on the proposed models. We show that the proposed method is able to generate realistic rtMRI video for unseen utterances, and adding CVAE is beneficial for learning the sequence-to-sequence mapping for subjects where the mapping is hard to learn.

show abstract

“…Details in Sections 2.1 and 3.1. 20,21,22]. Since these methods are data-driven and depend on the limited amount of articulatory data, they are unable to sufficiently generalize to unseen speakers to our knowledge.…”

Section: Introductionmentioning

confidence: 99%

Speaker-Independent Acoustic-to-Articulatory Speech Inversion

Wu¹,

Chen²,

Cho³

et al. 2023

Preprint

1

0

View full text Add to dashboard Cite

To build speech processing methods that can handle speech as naturally as humans, researchers have explored multiple ways of building an invertible mapping from speech to an interpretable space. The articulatory space is a promising inversion target, since this space captures the mechanics of speech production. To this end, we build an acoustic-toarticulatory inversion (AAI) model that leverages autoregression, adversarial training, and self supervision to generalize to unseen speakers. Our approach obtains 0.784 correlation on an electromagnetic articulography (EMA) dataset, improving the state-of-the-art by 12.5%. Additionally, we show the interpretability of these representations through directly comparing the behavior of estimated representations with speech production behavior. Finally, we propose a resynthesis-based AAI evaluation metric that does not rely on articulatory labels, demonstrating its efficacy with an 18-speaker dataset.

show abstract

Estimating articulatory movements in speech production with transformer networks

Cited by 3 publications

References 0 publications

Real-Time MRI Video Synthesis from Time Aligned Phonemes with Sequence-to-Sequence Networks

Real-Time MRI Video Synthesis from Time Aligned Phonemes with Sequence-to-Sequence Networks

Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networks

Speaker-Independent Acoustic-to-Articulatory Speech Inversion

Contact Info

Product

Resources

About