Autoencoder-Based Articulatory-to-Acoustic Mapping for Ultrasound Silent Speech Interfaces

Abstract

When using ultrasound video as input, Deep Neural Network-based Silent Speech Interfaces usually rely on the whole image to estimate the spectral parameters required for the speech synthesis step. Although this approach is quite straightforward, and it permits the synthesis of understandable speech, it has several disadvantages as well. Besides the inability to capture the relations between close regions (i.e. pixels) of the image, this pixelby-pixel representation of the image is also quite uneconomical. It is easy to see that a significant part of the image is irrelevant for the spectral parameter estimation task as the information stored by the neighbouring pixels is redundant, and the neural network is quite large due to the large number of input features. To resolve these issues, in this study we train an autoencoder neural network on the ultrasound image; the estimation of the spectral speech parameters is done by a second DNN, using the activations of the bottleneck layer of the autoencoder network as features. In our experiments, the proposed method proved to be more efficient than the standard approach: the measured normalized mean squared error scores were lower, while the correlation values were higher in each case. Based on the result of a listening test, the synthesized utterances also sounded more natural to native speakers. A further advantage of our proposed approach is that, due to the (relatively) small size of the bottleneck layer, we can utilize several consecutive ultrasound images during estimation without a significant increase in the network size, while significantly increasing the accuracy of parameter estimation.

Autoencoding Improves Pre-trained Word Embeddings

“…The connection between linear autoencoders and principal component analysis (PCA) was first proved by (Baldi and Hornik, 1989), which provides the basis for Theorem 2. The vast applications of autoencoders such as in language (Socher et al, 2011;Silberer and Lapata, 2014), speech (Gosztolya et al, 2019) and vision (Pu et al, 2016) domains suggest that non-linear autoencoders can indeed learn better representations than PCA.…”

Biosignal Sensors and Deep Learning-Based Speech Recognition: A Review

“…If the auditory signals cannot be used as input when recognizing the voice, the natural voice should be synthesized through the movement of the mobile organs of the vocal tract (e.g., tongue, lips). The system that performs automatic articulatory-to-acoustic mapping can be a component of these silent speech interfaces (SSIs) [ 6 ], and the studies are underway to solve the problems of not being able to record the voice signal itself [ 76 ] and reconstructing the voice by converting the articulatory movement of the voiceless patient into speech [ 77 ]. Table 2 presents a brief summary of the deep learning models so far.…”

“…Therefore, developing an effective algorithm to convert articulatory into speech is the main goal in SSI research [ 97 ], and deep learning technology has been introduced to achieve this goal. As the area of speech technology such as speech recognition and speech synthesis using deep learning has become wider, recent studies are attempting to solve the issue of articulatory-to-acoustic conversion [ 76 ]. In implementing SSI or silent speech recognition (SSR) technologies, such as sensor handling, interference, and feature extraction, using deep learning are also increasing to improve recognition performance [ 7 ].…”

“…In addition to the sensors(EMA, PMA, sEMG) introduced in Section 2 , the movement of articulators can be captured using various information, such as ultrasound tongue imaging (UTI), non-audible murmur (NAM), and the advantage of being able to collect voice information, especially from people who can’t make sounds, enables compensation for insufficient training data [ 77 ]. In addition, speech synthesis is used as an auxiliary technology for voice communication [ 77 ] as a process of inputting text and generating voice waveforms, with the introduction of deep learning in the speech technology area, it is possible to generate more natural voice [ 76 ]. So, using a large amount of motion data collected, people who have voice disorders can also synthesize with a natural voice.…”

Silent Speech Interfaces for Speech Restoration: A Review

“…This model achieved a recognition accuracy of 80.4% when tested over the database developed in [106], which validated it for visual speech recognition. Deep autoencoders were used in [273], [274] to extract features from ultrasound images, achieving significant gains in both silent ASR and direct synthesis. In [275], multitask learning of speech recognition and synthesis parameters was evaluated in the context of an ultrasoundbased SSI system designed to enhance the performance of individual tasks.…”