Data driven articulatory synthesis with deep neural networks

Aryal, Sandesh; Gutierrez‐Osuna, Ricardo

doi:10.1016/j.csl.2015.02.003

Cited by 39 publications

(26 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Current text-to-speech technologies have been able to produce speech with natural sounding voice for SSIs [19]. One of the current challenges of SSI development is silent speech recognition algorithms (without using audio data) [10,20] or mapping articulatory information to speech [21,22,23].…”

Section: Introductionmentioning

confidence: 99%

Determining an Optimal Set of Flesh Points on Tongue, Lips, and Jaw for Continuous Silent Speech Recognition

Wang¹,

Hahm²,

Mau³

2015

Proceedings of SLPAT 2015: 6th Workshop on Speech and Language Processing for Assistive Technologies

View full text Add to dashboard Cite

Articulatory data have gained increasing interest in speech recognition with or without acoustic data. Electromagnetic articulograph (EMA) is one of the affordable, currently used techniques for tracking the movement of flesh points on articulators (e.g., tongue) during speech. Determining an optimal set of sensors is important for optimizing the clinical applications of EMA data, due to the inconvenience of attaching sensors on tongue and other intraoral articulators, particularly for patients with neurological diseases. A recent study found an optimal set (tongue tip and body back, upper and lower lips) on tongue and lips for isolated phoneme, word, or short phrase classification from articulatory movement data. This four-sensor set, however, has not been verified in continuous silent speech recognition. In this paper, we investigated the use of data from sensor combinations in continuous speech recognition to verify the finding using a publicly available data set MOCHA-TIMIT. The long-standing speech recognition approach Gaussian mixture model (GMM)-hidden Markov model (HMM) and a recently available approach deep neural network (DNN)-HMM were used as the recognizers. Experimental results confirmed that the four-sensor set is optimal out of the full set of sensors on tongue, lips, and jaw. Adding upper incisor and/or velum data further improved the recognition performance slightly.

show abstract

Section: Introductionmentioning

confidence: 99%

Determining an Optimal Set of Flesh Points on Tongue, Lips, and Jaw for Continuous Silent Speech Recognition

Wang¹,

Hahm²,

Mau³

2015

Proceedings of SLPAT 2015: 6th Workshop on Speech and Language Processing for Assistive Technologies

View full text Add to dashboard Cite

show abstract

“…More details can be found in [50]. The choice for using a DNN-based mapping was motivated by previous results [35,51] showing that such mapping was more robust to noisy input than a state-of-the-art mapping based on Gaussian Mixture Model (GMM) such as the one proposed by Toda et al [34], and thus likely a better candidate for future BCI applications where input articulatory control signals will be inferred from noisy cortical signals.…”

Section: Methodsmentioning

confidence: 99%

Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces

et al. 2016

View full text Add to dashboard Cite

Restoring natural speech in paralyzed and aphasic people could be achieved using a Brain-Computer Interface (BCI) controlling a speech synthesizer in real-time. To reach this goal, a prerequisite is to develop a speech synthesizer producing intelligible speech in real-time with a reasonable number of control parameters. We present here an articulatory-based speech synthesizer that can be controlled in real-time for future BCI applications. This synthesizer converts movements of the main speech articulators (tongue, jaw, velum, and lips) into intelligible speech. The articulatory-to-acoustic mapping is performed using a deep neural network (DNN) trained on electromagnetic articulography (EMA) data recorded on a reference speaker synchronously with the produced speech signal. This DNN is then used in both offline and online modes to map the position of sensors glued on different speech articulators into acoustic parameters that are further converted into an audio signal using a vocoder. In offline mode, highly intelligible speech could be obtained as assessed by perceptual evaluation performed by 12 listeners. Then, to anticipate future BCI applications, we further assessed the real-time control of the synthesizer by both the reference speaker and new speakers, in a closed-loop paradigm using EMA data recorded in real time. A short calibration period was used to compensate for differences in sensor positions and articulatory differences between new speakers and the reference speaker. We found that real-time synthesis of vowels and consonants was possible with good intelligibility. In conclusion, these results open to future speech BCI applications using such articulatory-based speech synthesizer.

show abstract

“…In addition to the past and current source vectors, we also consider for the mapping ω future source vectors as this often improves the mapping accuracy at the expense of introducing a small delay in the conversion process [4,5,13]. Provided that this delay is less than 50 ms, we will be able to restore the articulatory-auditory feedback without causing disfluencies or mental stress on the speaker [14,15].…”

Section: Speech Synthesis From Articulator Movementmentioning

confidence: 99%

“…For comparison purposes, we also evaluate mappings using GMMs [4,25,26] and DNNs [13,27], which have been successfully applied by ourselves and other authors to model the articulatory-to-acoustic mapping. For a fair comparison, GMMs and DNNs with approximately the same number of parameters as the RNN architecture (˜1/2 million parameters) are employed.…”

Section: Model Trainingmentioning

confidence: 99%

Interspeech 2017

2017

View full text Add to dashboard Cite

To help people who have lost their voice following total laryngectomy, we present a speech restoration system that produces audible speech from articulator movement. The speech articulators are monitored by sensing changes in magnetic field caused by movements of small magnets attached to the lips and tongue. Then, articulator movement is mapped to a sequence of speech parameter vectors using a transformation learned from simultaneous recordings of speech and articulatory data. In this work, this transformation is performed using a type of recurrent neural network (RNN) with fixed latency, which is suitable for realtime processing. The system is evaluated on a phoneticallyrich database with simultaneous recordings of speech and articulatory data made by non-impaired subjects. Experimental results show that our RNN-based mapping obtains more accurate speech reconstructions (evaluated using objective quality metrics and a listening test) than articulatory-to-acoustic mappings using Gaussian mixture models (GMMs) or deep neural networks (DNNs). Moreover, our fixed-latency RNN architecture provides comparable performance to an utterance-level batch mapping using bidirectional RNNs (BiRNNs).

show abstract

Data driven articulatory synthesis with deep neural networks

Cited by 39 publications

References 33 publications

Determining an Optimal Set of Flesh Points on Tongue, Lips, and Jaw for Continuous Silent Speech Recognition

Determining an Optimal Set of Flesh Points on Tongue, Lips, and Jaw for Continuous Silent Speech Recognition

Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces

Interspeech 2017

Contact Info

Product

Resources

About