Direct Speech Reconstruction From Articulatory Sensor Data by Machine Learning

Abstract

This paper describes a technique which generates speech acoustics from articulator movements. Our motivation is to help people who can no longer speak following laryngectomy, a procedure which is carried out tens of thousands of times per year in the Western world. Our method for sensing articulator movement, Permanent Magnetic Articulography, relies on small, unobtrusive magnets attached to the lips and tongue. Changes in magnetic field caused by magnet movements are sensed and form the input to a process which is trained to estimate speech acoustics. In the experiments reported here this 'Direct Synthesis' technique is developed for normal speakers, with glued-on magnets, allowing us to train with parallel sensor and acoustic data. We describe three machine learning techniques for this task, based on Gaussian Mixture Models (GMMs), Deep Neural Networks (DNNs) and Recurrent Neural Networks (RNNs). We evaluate our techniques with objective acoustic distortion measures and subjective listening tests over spoken sentences read from novels (the CMU Arctic corpus). Our results show that the best performing technique is a bidirectional RNN (BiRNN), which employs both past and future contexts to predict the acoustics from the sensor data. BiRNNs are not suitable for synthesis in real-time but fixed-lag RNNs give similar results and, because they only look a little way into the future, overcome this problem. Listening tests show that the speech produced by this method has a natural quality which preserves the identity of the speaker. Furthermore, we obtain up to 92% intelligibility on the challenging CMU Arctic material. To our knowledge, these are the best results obtained for a silent-speech system without a restricted vocabulary and with an unobtrusive device that delivers audio in close to real time. This work promises to lead to a technology which truly will give people whose larynx has been removed their voices back.

Ultrasound-Based Silent Speech Interface Built on a Continuous Vocoder

“…There are two distinct ways of SSI solutions, namely 'direct synthesis' and 'recognition-and-synthesis' [21]. In the first case, the speech signal is generated without an intermediate step, directly from the articulatory data, typically using vocoders [4,5,6,8,9,10,11,15,17]. In the second case, silent speech recognition (SSR) is applied on the biosignal which extracts the content spoken by the person (i.e.…”

DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging

“…In another paper a multimodal Deep AutoEncoder was used to synthesize sung vowels based on ultrasound recordings and a video of the lips [27]. Gonzalez and his colleagues compared GMM, DNN and RNN [36] for PMA-based direct synthesis, Csapó et al used DNNs to predict the spectral parameters [6] and F0 [28] of a vocoder using UTI as articulatory input. For the prediction of the V/U flag and F0 using articulatory input, multiple DNN architectures were compared, including DNN, RNN and LSTM neural networks [46], [47].…”

Speech Classification using Logical ART Deep Mechanism of Machine Learning

“…The use of learning models is very much relevant and learning methods like SVM is widely used due their function characteristic of non-linearity to map the data to a very high dimension but at the same time it is not effective to handle many properties of dataset of speech emotion. The authors (C. Zha, P. Yang, X. Zhang and L. Zhao,2016) handles this issue of uni-functionality by introducing multiple kernels and tested on the Aibo dataset which exhibits better result as compared to the SVM [9].…”