Toward Silent Paralinguistics: Speech-to-EMG — Retrieving Articulatory Muscle Activity from Speech

Abstract

Electromyographic (EMG) signals recorded during speech production encode information on articulatory muscle activity and also on the facial expression of emotion, thus representing a speech-related biosignal with strong potential for paralinguistic applications. In this work, we estimate the electrical activity of the muscles responsible for speech articulation directly from the speech signal. To this end, we first perform a neural conversion of speech features into electromyographic time domain features, and then attempt to retrieve the original EMG signal from the time domain features. We propose a feed forward neural network to address the first step of the problem (speech features to EMG features) and a neural network composed of a convolutional block and a bidirectional long short-term memory block to address the second problem (true EMG features to EMG signal). We observe that four out of the five originally proposed time domain features can be estimated reasonably well from the speech signal. Further, the five time domain features are able to predict the original speech-related EMG signal with a concordance correlation coefficient of 0.663. We further compare our results with the ones achieved on the inverse problem of generating acoustic speech features from EMG features.

Towards Silent Paralinguistics: Deriving Speaking Mode and Speaker ID from Electromyographic Signals

“…We further calculate the concordance correlation coefficient (CCC) for each of the MFCCs, as estimated from the EMG data versus from the reference audio (Figure 2), suggesting that lower coefficients (i. e., large spectral changes) were predicted relatively well, whereas the prediction quality for the higher coefficients was degraded. As demonstrated by our complementary work on retrieving articulatory muscle activity from EMG [19], the CCC for this inverse SCP problem is around 0.62. I. e., results for Speech-to-EMG CCCs have been comparable to the prediction of the first two MFCCs in figure 2.…”

Künstliche Intelligenz und Ethik im Gesundheitswesen – Spagat oder Symbiose?

“…Hier könnten insbesondere biosignalbasierte adaptive Systeme [26] einen Beitrag leisten, indem sie z. B. hel-fen, die verbleibenden sprachlichen und kommunikativen Fähigkeiten einer er-kranktenPersonzu unterstützen [27][28][29][30]. Ebenso könnten diese Systeme lernen zu erkennen, durch welche Angebote und Anregungen die an Demenz erkrankte Person wieder zu mehr aktiver Teilnahme (englisch: "Engagement") am Alltagsgeschehen angeregt werden könnte [31][32][33][34].…”