Guided Learning of Pronunciation by Visualizing Tongue Articulation in Ultrasound Image Sequences

Mozaffari, Mohammad; Guan, Shenyong; Wen, Shuangyue; Wang, Nan; Lee, Won-Sook

doi:10.1109/civemsa.2018.8440000

Cited by 14 publications

(11 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Beyond speech therapy, ultrasound is used in phonetics research to compare tongue-shapes for different phones (Davidson, 2006;Lee-Kim et al, 2014;Chen & Lin, 2011;Lawson et al, 2015;Ahn, 2018), or to gain insight into speech production through articulatory-to-acoustic or acoustic-to-articulatory mapping (Hueber et al, 2011;Porras et al, 2019). Ultrasound is also used for practical tasks such as second language learning and acquisition (Wilson et al, 2006;Gick et al, 2008;Mozaffari et al, 2018), or to drive silent speech interfaces, which can be used to restore spoken communication for users with voice impairments (Denby & Stone, 2004;Hueber et al, 2010;Csapó et al, 2017;Ji et al, 2018;Ribeiro et al, 2021b).…”

Section: Ultrasound Tongue Imagingmentioning

confidence: 99%

Automatic audiovisual synchronisation for ultrasound tongue imaging

Eshky,

Cleland,

Ribeiro

et al. 2021

Preprint

View full text Add to dashboard Cite

Ultrasound tongue imaging is used to visualise the intra-oral articulators during speech production. It is utilised in a range of applications, including speech and language therapy and phonetics research. Ultrasound and speech audio are recorded simultaneously, and in order to correctly use this data, the two modalities should be correctly synchronised. Synchronisation is achieved using specialised hardware at recording time, but this approach can fail in practice resulting in data of limited usability. In this paper, we address the problem of automatically synchronising ultrasound and audio after data collection. We first investigate the tolerance of expert ultrasound users to synchronisation errors in order to find the thresholds for error detection. We use these thresholds to define accuracy scoring boundaries for evaluating our system. We then describe our approach for automatic synchronisation, which is driven by a self-supervised neural network, exploiting the correlation between the two signals to synchronise them. We train our model on data from multiple domains with different speaker characteristics, different equipment, and different recording environments, and achieve an accuracy >92.4% on held-out in-domain data. Finally, we introduce a novel resource, the Cleft dataset, which we gathered with a new clinical subgroup and for which hardware synchronisation proved unreliable. We apply our model to this out-of-domain data, and evaluate its performance subjectively with expert users. Results show that users prefer our model's output over the original hardware output 79.3% of the time. Our results demonstrate the strength of our approach and its ability to generalise to data from new domains.

show abstract

Section: Ultrasound Tongue Imagingmentioning

confidence: 99%

Automatic audiovisual synchronisation for ultrasound tongue imaging

Eshky,

Cleland,

Ribeiro

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, it is crucial to have a fully automatic system for the tongue contour extraction. It is even harder for the case of tongue contour tracking in real-time applications (Mozaffari et al, 2018). Various methods have been utilized for the problem of automatic tongue extraction in the last recent years such as image segmentation like active contour models or snakes (Ghrenassia et al, 2014;Laporte and Ménard, 2015;Li et al, 2005;Xu et al, 2016bXu et al, , 2016c, graph-based technique (Tang and Hamarneh, 2010), machine learning-based methods (Berry and Fasel, 2011;Fabre et al, 2015;Fasel and Berry, 2010;L.…”

Section: Literature Review and Related Workmentioning

confidence: 99%

“…Proposing two robust, accurate, fully-automatic deep learning architectures specifically for tongue contour extraction while the proposed models have the capability of real-time performance using GPU power. An alternative solution for CPU only purposes has been reported in previous research (Mozaffari et al, 2018). For the first time, dilated convolution is combined with the standard deep, dense classification method in two new network architectures to extract both local and global context from each frame at the same time in an end-to-end fashion.…”

mentioning

confidence: 99%

BowNet: Dilated convolutional neural network for ultrasound tongue contour extraction

Mozaffari

Lee²

2019

The Journal of the Acoustical Society of America

Self Cite

View full text Add to dashboard Cite

Ultrasound imaging is safe, relatively affordable, and capable of real-time performance. This technology has been used for real-time visualization and analyzing the functionality of human organs in many studies. One application of this technology is to visualize and to characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the noisy nature of ultrasound images with low-contrast characteristic, it might require expertise for non-expert users to recognize organ shape such as tongue surface (dorsum). To alleviate this difficulty for quantitative analysis of tongue shape and motion, tongue surface can be extracted, tracked, and visualized instead of the whole tongue region. Delineating the tongue surface from each frame is a cumbersome, subjective, and error-prone task. Furthermore, the rapidity and complexity of tongue gestures have made it a challenging task, and manual segmentation is not a feasible solution for real-time applications. The progress of deep convolutional neural networks has been successfully exploited in various computer vision tasks such as image classification and segmentation. Several end-to-end deep learning segmentation methods provide a promising alternative for previous techniques with higher accuracy and robustness results, without any intervention. Employing the power of highspeed graphics processing unit (GPU) with state-of-the-art deep neural network models and training techniques, it is feasible to implement new fully-automatic, accurate, and robust segmentation methods with the capability of real-time performance, applicable for tracking of the tongue contours during the speech. This paper presents two novel deep neural network models named BowNet and wBowNet benefits from the ability of global prediction of decoding-encoding models, with integrated multi-scale contextual information, and capability of full-resolution (local) extraction of dilated convolutions. Experimental results using several ultrasound tongue image datasets revealed that the combination of both localization and globalization searching could improve prediction result significantly. Assessment of BowNet models using both qualitatively and quantitatively studies showed their outstanding achievements in terms of accuracy and robustness in compare with similar techniques.

show abstract

“…End-to-end fashion supervised deep learning techniques, outperformed previous techniques in recent years. For example, U-net [12] has been used for automatic ultrasound tongue extraction [18], [9]. After successful results of deep learning methods, the focus of advanced techniques for tongue contour extraction is more on generalization and real-time performance [5], [10], [11].…”

Section: Introductionmentioning

confidence: 99%

Deep Learning for Automatic Tracking of Tongue Surface in Real-time Ultrasound Videos, Landmarks instead of Contours

Mozaffari¹,

Lee²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

One usage of medical ultrasound imaging is to visualize and characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the low-contrast characteristic and noisy nature of ultrasound images, it might require expertise for non-expert users to recognize tongue gestures in applications such as visual training of a second language. Moreover, quantitative analysis of tongue motion needs the tongue dorsum contour to be extracted, tracked, and visualized. Manual tongue contour extraction is a cumbersome, subjective, and error-prone task. Furthermore, it is not a feasible solution for real-time applications. The growth of deep learning has been vigorously exploited in various computer vision tasks, including ultrasound tongue contour tracking. In the current methods, the process of tongue contour extraction comprises two steps of image segmentation and post-processing. This paper presents a new novel approach of automatic and real-time tongue contour tracking using deep neural networks. In the proposed method, instead of the two-step procedure, landmarks of the tongue surface are tracked. This novel idea enables researchers in this filed to benefits from available previously annotated databases to achieve high accuracy results. Our experiment disclosed the outstanding performances of the proposed technique in terms of generalization, performance, and accuracy.

show abstract

Guided Learning of Pronunciation by Visualizing Tongue Articulation in Ultrasound Image Sequences

Cited by 14 publications

References 18 publications

Automatic audiovisual synchronisation for ultrasound tongue imaging

Automatic audiovisual synchronisation for ultrasound tongue imaging

BowNet: Dilated convolutional neural network for ultrasound tongue contour extraction

Deep Learning for Automatic Tracking of Tongue Surface in Real-time Ultrasound Videos, Landmarks instead of Contours

Contact Info

Product

Resources

About