3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces

Tóth, László; Shandiz, Amin Honarmandi

doi:10.1007/978-3-030-61401-0_16

Cited by 8 publications

(20 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This parameter allows us to analyze input blocks that are placed at bigger time intervals. In an earlier study we processed ultrasound videos which had a much larger frame rate, and the optimal value for sts was found to be 5 [32]. Here, we got the best performance with sts = 3, which is reasonable as the frame rate was much lower.…”

Section: D-cnn+bilstmsupporting

confidence: 55%

“…Formally, our networks has to map each MRI image to a spectral vector. However, using several consecutive input frames instead of a single frame can significantly improve the results [6], [32]. Hence, the input for all our network configurations was a 3D array, treating time as the the third axis besides the two spacial axes of the images.…”

Section: D-cnn+bilstmmentioning

confidence: 99%

“…3D-CNN: Several authors argued recently that good video classification is also achievable using purely convolutional structures by extending the convolution to the temporal axis [6], [32], [36]. This is why a 3D convolutional model served as our second model (see Table I).…”

Section: D-cnn+bilstmmentioning

confidence: 99%

See 2 more Smart Citations

Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders

Yu¹,

Shandiz²,

Tóth³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Several approaches exist for the recording of articulatory movements, such as eletromagnetic and permanent magnetic articulagraphy, ultrasound tongue imaging and surface electromyography. Although magnetic resonance imaging (MRI) is more costly than the above approaches, the recent developments in this area now allow the recording of real-time MRI videos of the articulators with an acceptable resolution.Here, we experiment with the reconstruction of the speech signal from a real-time MRI recording using deep neural networks. Instead of estimating speech directly, our networks are trained to output a spectral vector, from which we reconstruct the speech signal using the WaveGlow neural vocoder. We compare the performance of three deep neural architectures for the estimation task, combining convolutional (CNN) and recurrence-based (LSTM) neural layers. Besides the mean absolute error (MAE) of our networks, we also evaluate our models by comparing the speech signals obtained using several objective speech quality metrics like the mean cepstral distortion (MCD), Short-Time Objective Intelligibility (STOI), Perceptual Evaluation of Speech Quality (PESQ) and Signal-to-Distortion Ratio (SDR). The results indicate that our approach can successfully reconstruct the gross spectral shape, but more improvements are needed to reproduce the fine spectral details.

show abstract

Section: D-cnn+bilstmsupporting

confidence: 55%

Section: D-cnn+bilstmmentioning

confidence: 99%

See 1 more Smart Citation

Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders

Yu¹,

Shandiz²,

Tóth³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The input of our network is a 3D array of consecutive images, and the output is a 80-dimensional spectral vector. The convolutional (3D-CNN) network structure that we applied here [30] was the same as the lower, 'frame-level' part of the x-vector network, so we delay its presentation to the next section. As here the task is to estimate spectral vectors, we used a linear output layer and the network was trained to minimize the mean-squared error (MSE) of the regression task.…”

Section: The Ssi Frameworkmentioning

confidence: 99%

Neural Speaker Embeddings for Ultrasound-Based Silent Speech Interfaces

et al. 2021

Self Cite

View full text Add to dashboard Cite

Articulatory-to-acoustic mapping seeks to reconstruct speech from a recording of the articulatory movements, for example, an ultrasound video. Just like speech signals, these recordings represent not only the linguistic content, but are also highly specific to the actual speaker. Hence, due to the lack of multi-speaker data sets, researchers have so far concentrated on speaker-dependent modeling. Here, we present multi-speaker experiments using the recently published TaL80 corpus. To model speaker characteristics, we adjusted the x-vector framework popular in speech processing to operate with ultrasound tongue videos. Next, we performed speaker recognition experiments using 50 speakers from the corpus. Then, we created speaker embedding vectors and evaluated them on the remaining speakers. Finally, we examined how the embedding vector influences the accuracy of our ultrasound-to-speech conversion network in a multi-speaker scenario. In the experiments we attained speaker recognition error rates below 3%, and we also found that the embedding vectors generalize nicely to unseen speakers. Our first attempt to apply them in a multi-speaker silent speech framework brought about a marginal reduction in the error rate of the spectral estimation step.

show abstract

“…Here, we experiment with two neural network configurations in our SSI framework, that is, to estimate a speech mel-spectrogram from a sequence of ultrasound images. The first network is a 3D-CNN, following the proposal of Tóth et al [24]. The second configuration combines the 3D-CNN layers with and additional BiLSTM layer, as it may be more effective in aggregating the information along the time axis.…”

Section: Cnns For the Ssi And For The Vad Taskmentioning

confidence: 99%

Voice Activity Detection for Ultrasound-Based Silent Speech Interfaces Using Convolutional Neural Networks

Shandiz¹,

Tóth²

2021

Text, Speech, and Dialogue

Self Cite

View full text Add to dashboard Cite

Voice Activity Detection (VAD) is not easy task when the input audio signal is noisy, and it is even more complicated when the input is not even an audio recording. This is the case with Silent Speech Interfaces (SSI) where we record the movement of the articulatory organs during speech, and we aim to reconstruct the speech signal from this recording. Our SSI system synthesizes speech from ultrasonic videos of the tongue movement, and the quality of the resulting speech signals are evaluated by metrics such as the mean squared error loss function of the underlying neural network and the Mel-Cepstral Distortion (MCD) of the reconstructed speech compared to the original. Here, we first demonstrate that the amount of silence in the training data can have an influence both on the MCD evaluation metric and on the performance of the neural network model. Then, we train a convolutional neural network classifier to separate silent and speech-containing ultrasound tongue images, using a conventional VAD algorithm to create the training labels from the corresponding speech signal. In the experiments our ultrasound-based speech/silence separator achieved a classification accuracy of about 85% and an AUC score around 86%.

show abstract

3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces

Cited by 8 publications

References 32 publications

Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders

Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders

Neural Speaker Embeddings for Ultrasound-Based Silent Speech Interfaces

Voice Activity Detection for Ultrasound-Based Silent Speech Interfaces Using Convolutional Neural Networks

Contact Info

Product

Resources

About