Articulation-to-Speech Synthesis Using Articulatory Flesh Point Sensors’ Orientation Information

Cao, Beiming; Kim, Myung Jong; Wang, Jun R.; Santen, Jan P. H. van; Mau, Ted; Wang, Jun

doi:10.21437/interspeech.2018-2484

Cited by 20 publications

(22 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This has the main idea of recording the soundless articulatory movement, and automatically generating speech from the movement information, while the subject is not producing any sound. For this automatic conversion task, typically electromagnetic articulography (EMA) [2,3,4,5], ultrasound tongue imaging (UTI) [6,7,8,9,10,11,12,13], permanent magnetic articulography (PMA) [14,15], surface electromyography (sEMG) [16,17,18], Non-Audible Murmur (NAM) [19] or video of the lip movements [7,20] are used.…”

Section: Introductionmentioning

confidence: 99%

“…There are two distinct ways of SSI solutions, namely 'direct synthesis' and 'recognition-and-synthesis' [21]. In the first case, the speech signal is generated without an intermediate step, directly from the articulatory data, typically using vocoders [4,5,6,8,9,10,11,15,17]. In the second case, silent speech recognition (SSR) is applied on the biosignal which extracts the content spoken by the person (i.e.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ultrasound-Based Silent Speech Interface Built on a Continuous Vocoder

et al. 2019

View full text Add to dashboard Cite

Recently it was shown that within the Silent Speech Interface (SSI) field, the prediction of F0 is possible from Ultrasound Tongue Images (UTI) as the articulatory input, using Deep Neural Networks for articulatory-to-acoustic mapping. Moreover, text-to-speech synthesizers were shown to produce higher quality speech when using a continuous pitch estimate, which takes non-zero pitch values even when voicing is not present. Therefore, in this paper on UTI-based SSI, we use a simple continuous F0 tracker which does not apply a strict voiced / unvoiced decision. Continuous vocoder parameters (ContF0, Maximum Voiced Frequency and Mel-Generalized Cepstrum) are predicted using a convolutional neural network, with UTI as input. The results demonstrate that during the articulatory-toacoustic mapping experiments, the continuous F0 is predicted with lower error, and the continuous vocoder produces slightly more natural synthesized speech than the baseline vocoder using standard discontinuous F0.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Ultrasound-Based Silent Speech Interface Built on a Continuous Vocoder

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Furthermore, direct speech generation from the captured biosignals (see Section III-B) is another possibility, having this approach the potential to restore the person's own voice, if enough recordings of the pre-laryngectomy voice are available for training [96]. This second approach has been also validated for various modalities, including sEMG [31], [97]- [100], PMA [25], [96], [101]- [103], video-and-ultrasound [104]- [106] and Doppler signals [107]. To sum up, the foundations have been laid for a future SSI-based device for postlaryngectomy speech rehabilitation.…”

Section: Voice Disordersmentioning

confidence: 99%

Silent Speech Interfaces for Speech Restoration: A Review

et al. 2020

View full text Add to dashboard Cite

This review summarises the status of silent speech interface (SSI) research. SSIs rely on non-acoustic biosignals generated by the human body during speech production to enable communication whenever normal verbal communication is not possible or not desirable. In this review, we focus on the first case and present latest SSI research aimed at providing new alternative and augmentative communication methods for persons with severe speech disorders. SSIs can employ a variety of biosignals to enable silent communication, such as electrophysiological recordings of neural activity, electromyographic (EMG) recordings of vocal tract movements or the direct tracking of articulator movements using imaging techniques. Depending on the disorder, some sensing techniques may be better suited than others to capture speech-related information. For instance, EMG and imaging techniques are well suited for laryngectomised patients, whose vocal tract remains almost intact but are unable to speak after the removal of the vocal folds, but fail for severely paralysed individuals. From the biosignals, SSIs decode the intended message, using automatic speech recognition or speech synthesis algorithms. Despite considerable advances in recent years, most present-day SSIs have only been validated in laboratory settings for healthy users. Thus, as discussed in this paper, a number of challenges remain to be addressed in future research before SSIs can be promoted to real-world applications. If these issues can be addressed successfully, future SSIs will improve the lives of persons with severe speech impairments by restoring their communication capabilities.

show abstract

“…Biosignal-based speech communication has shown increasing promise towards various clinical applications [1] such as silent speech interface (SSI), which directly converts non-audio articulatory information to speech to help individuals who have lost their ability of speech production but can still articulate silently (e.g., laryngectomees) [2]. Besides novel tongue and lip motion tracking devices, current SSI research is focused on developing algorithms that can directly map the articulatory information to speech (text/acoustics) accurately and efficiently [2][3][4][5][6].…”

Section: Introductionmentioning

confidence: 99%

Towards a Speaker Independent Speech-BCI Using Speaker Adaptation

et al. 2019

Self Cite

View full text Add to dashboard Cite

Neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS) can cause locked-in-syndrome (fully paralyzed but aware). Brain-computer interface (BCI) may be the only option to restore their communication. Current BCIs typically use visual or attention correlates in neural activities to select letters randomly displayed on a screen, which are extremely slow (a few words per minute). Speech-BCIs, which aim to convert the brain activity patterns to speech (neural speech decoding), hold the potential to enable faster communication. Although a few recent studies have shown the potential of neural speech decoding, those are focused on speaker-dependent models. In this study, we investigated speaker-independent neural speech decoding of five continuous phrases from Magnetoencephalography (MEG) signals while 8 subjects produced speech covertly (imagination) or overtly (articulation). We have used both supervised and unsupervised speaker adaptation strategies for implementing a speaker independent model. Experimental results demonstrated that the proposed adaptation-based speakerindependent model has significantly improved decoding performance. To our knowledge, this is the first demonstration of the possibility of speaker-independent neural speech decoding.

show abstract

Articulation-to-Speech Synthesis Using Articulatory Flesh Point Sensors’ Orientation Information

Cited by 20 publications

References 14 publications

Ultrasound-Based Silent Speech Interface Built on a Continuous Vocoder

Ultrasound-Based Silent Speech Interface Built on a Continuous Vocoder

Silent Speech Interfaces for Speech Restoration: A Review

Towards a Speaker Independent Speech-BCI Using Speaker Adaptation

Contact Info

Product

Resources

About