An initial investigation into the real-time conversion of facial surface EMG signals to audible speech

Abstract

This paper presents early-stage results of our investigations into the direct conversion of facial surface electromyographic (EMG) signals into audible speech in a real-time setting, enabling novel avenues for research and system improvement through real-time feedback. The system uses a pipeline approach to enable online acquisition of EMG data, extraction of EMG features, mapping of EMG features to audio features, synthesis of audio waveforms from audio features and output of the audio waveforms via speakers or headphones. Our system allows for performing EMG-to-Speech conversion with low latency and on a continuous stream of EMG data, enabling near instantaneous audio output during audible as well as silent speech production. In this paper, we present an analysis of our systems components for latency incurred, as well as the tradeoffs between conversion quality, latency and training duration required.

Direct Speech Reconstruction From Articulatory Sensor Data by Machine Learning

“…DNNs have been shown to achieve state-of-the-art performance in various speech tasks such as automatic speech recognition [34], speech synthesis [35], [36], voice conversion [37] and articulatory-to-acoustic mapping [6], [19]- [21] as well. Inspired by this, we describe in this section an alternative technique which employs DNNs to represent the mapping between PMA and speech feature vectors.…”

“…A silent speech interface (SSI) [1] is a system that provides this form of silent communication automatically. The principle of an SSI is that the speech that a person wishes to produce can be inferred from non-acoustic sources of information generated during speech articulation, such as the brain's electrical activity [2], [3], the electrical activity produced by the articulator muscles [4]- [6] or the movement of the speech articulators [7]- [10]. In the past few years there has been an increased interest among the scientific community in SSIs due to their potential applications.…”

“…The availability of devices for capturing articulatory data along with improvements in supervised machine learning techniques have made datadriven methods more competitive than model-based methods in terms of mapping accuracy. To learn the articulatory-toacoustic mapping from parallel data, several machine learning techniques have been investigated such as Gaussian mixture models (GMMs) [8], [17], hidden Markov models (HMMs) [18] and neural networks [6], [10], [19]- [21].…”

“…This will be our baseline mapping system. Then, encouraged by the recent success of deep neural networks (DNNs) in several speech applications, including its application by other authors [6], [10], [19]- [21] to model the articulatory-to-acoustic mapping for other SSIs, we also investigate here the application of DNNs to synthesise speech from PMA data. In comparison with previous work, in this paper we carry out an extensive evaluation on the effect on the speech quality generated by the GMM and DNN mapping approaches when using the following features in the mapping: (i) segmental, contextual features computed by concatenating several PMA samples to capture the articulator dynamics, (ii) the maximum likelihood parameter generations (MLPGs) algorithm [26], [27] to obtain smoother temporal trajectories for the predicted speech features, and (iii) conversion considering the global variance (GV) of the speech features, which has been shown to improve the perceived quality for speech synthesis and voice conversion (VC), but have not been extensively investigated for articulatory-to-speech conversion.…”

Investigating Objective Intelligibility in Real-Time EMG-to-Speech Conversion

“…In our previous work, we have described a basic neural network based EMG-to-Speech conversion system, performing offline conversion [13,10], as well as a first real time system [14]. While original analysis of the real-time system focused on timing performance, this paper will examine the impact of different parameters on output quality using small amounts of training data.…”