Vocal Tract Contour Tracking in rtMRI Using Deep Temporal Regression Network

Abstract

Recent advances in real-time Magnetic Resonance Imaging (rtMRI) provide an invaluable tool to study speech articulation. In this paper, we present an effective deep learning approach for supervised detection and tracking of vocal tract contours in a sequence of rtMRI frames. We train a single input multiple output deep temporal regression network (DTRN) to detect the vocal tract (VT) contour and the separation boundary between different articulators. The DTRN learns the non-linear mapping from an overlapping fixed-length sequence of rtMRI frames to the corresponding articulatory movements, where a blend of the overlapping contour estimates defines the detected VT contour. The detected contour is refined at a post-processing stage using an appearance model to further improve the accuracy of VT contour detection. The proposed VT contour tracking model is trained and evaluated over the USC-TIMIT dataset. Performance evaluation is carried out using three objective assessment metrics for the separating landmark detection, contour tracking and temporal stability of the contour landmarks in comparison with three baseline approaches from the recent literature. Results indicate significant improvements with the proposed method over the state-of-the-art baselines.

An open-source toolbox for measuring vocal tract shape from real-time magnetic resonance images

“…Previous methodologies have implemented greater automation by leveraging deep learning techniques. However, these rely on training data sourced from a relatively narrow sample of behaviours, imaging centres, and pulse sequences (e.g., Asadiabadi & Erzin, 2020 ; Bresch & Narayanan, 2009 ; Eslami et al, 2020 ; Labrunie et al, 2018 ; van Leeuwen et al, 2019 ; Mannem & Ghosh, 2021 ; Pandey & Sabbir Arif, 2021 ; Ruthven et al, 2021 ; Silva & Teixeira, 2015 ; Somandepalli et al, 2017 ; Takemoto et al, 2019 ; Valliappan et al, 2019 ) . While our pipeline is more labour-intensive, we have demonstrated that it generalises beyond the dataset for which it was designed (see Belyk et al, 2022 ).…”

“…A promising avenue for development has come from the application of machine learning, and deep learning algorithms in particular (Goodfellow et al, 2016 ). This is a field of statistics that continues to develop rapidly, and several implementations have been proposed for applications to rtMRI (e.g., Asadiabadi & Erzin, 2020 ; Bresch & Narayanan, 2009 ; Eslami et al, 2020 ; Labrunie et al, 2018 ; van Leeuwen et al, 2019 ; Mannem & Ghosh, 2021 ; Pandey & Sabbir Arif, 2021 ; Ruthven et al, 2021 ; Silva & Teixeira, 2015 ; Somandepalli et al, 2017 ; Takemoto et al, 2019 ; Valliappan et al, 2019 ). Deep-learning-based approaches can yield machine-generated traces of the vocal tract which are sufficiently accurate to be useful for scientific measurements.…”

An open-source toolbox for measuring vocal tract shape from real-time magnetic resonance images

“…Previous methodologies have implemented greater automation by leveraging powerful machine learning techniques. However, these make strong assumptions about the underlying data or rely on models which are trained from a narrow sample which reduces their generalisability to new research [12][13][14][15][16]. We have demonstrated that our pipeline generalises beyond the dataset for which it was designed.…”

“…Furthermore, those approaches which are available have not yet been demonstrated to generalise beyond the individual datasets for which they were developed. These methodologies typically involve automated or machine learning processes which are trained and tested against a narrow range of data, typically composed of a small number of speakers scanned at a single imaging centre [12][13][14][15][16][17][18]. The development of these techniques has sampled disproportionally from a single image repository [6].…”

Automatic generation of the complete vocal tract shape from the sequence of phonemes to be articulated

“…Initially proposed for tracking the tongue contour in ultrasound images with autoencoders [31], these techniques have been used for MRI images with the help of U-Net [32] as we did [21]. Other methods like Deep Temporal Regression Network (DTRN), which take into account a series of images, present the advantage of integrating the movement for tracking several articulators [33].…”