Investigating Objective Intelligibility in Real-Time EMG-to-Speech Conversion

Abstract

This paper presents an analysis of the influence of various system parameters on the output quality of our neural network based real-time EMG-to-Speech conversion system. This EMG-to-Speech system allows for the direct conversion of facial surface electromyographic signals into audible speech in real time, allowing for a closed-loop setup where users get direct audio feedback. Such a setup opens new avenues for research and applications through co-adaptation approaches. In this paper, we evaluate the influence of several parameters on the output quality, such as time context, EMG-Audio delay, network-, training data-and Mel spectrogram size. The resulting output quality is evaluated based on the objective output quality measure STOI.

Silent Speech Interfaces for Speech Restoration: A Review

“…These low values can only be achieved through direct speech synthesis. In this sense, real-time SSI systems have been developed for sEMG [181], [182], PMA [183] and EMA [27]. There is also the possibility that real-time auditory feedback might enable the brain to assimilate the SSI as if it were the person's own voice, thus enabling the user to adapt her/his own speaking patterns to produce better acoustics.…”

“…Direct speech synthesis from EMG signals has also progressed considerably in recent years (see [31], [99], [181], [182]), following advances in array sEMG sensors and deep learning. As mentioned above, a particular advantage of EMG with respect to other techniques for articulator motion capture is that EMG signals can be sensed~60 ms before the actual movements of the articulators.…”

“…As mentioned above, a particular advantage of EMG with respect to other techniques for articulator motion capture is that EMG signals can be sensed~60 ms before the actual movements of the articulators. This rapidity facilitates the development of real-time direct synthesis systems with low latency [181], [182], so that the delay between the articulatory gestures and the synthesised acoustic feedback is minimal. In [182], a comprehensive study was carried out in which the influence of various system parameters (DNN size, amount of training data, frame shift, etc.)…”

“…This rapidity facilitates the development of real-time direct synthesis systems with low latency [181], [182], so that the delay between the articulatory gestures and the synthesised acoustic feedback is minimal. In [182], a comprehensive study was carried out in which the influence of various system parameters (DNN size, amount of training data, frame shift, etc.) on the speech quality generated by a real-time direct synthesis system was analysed using objective quality metrics.…”

Applying DNN Adaptation to Reduce the Session Dependency of Ultrasound Tongue Imaging-based Silent Speech Interfaces

“…extremely noisy environments and/or military situations). For this automatic conversion task, typically electromagnetic articulography (EMA, [3,19,20]), ultrasound tongue imaging (UTI, [4,14,18,28]), permanent magnetic articulography (PMA, [10]), surface Electromyography (sEMG, [6,16,22]), lip video [1,7] and multimodal approaches are used [5]. Current SSI systems mostly apply the "direct synthesis" principle, where speech is generated without an intermediate step, directly from the articulatory data.…”