Improving Fundamental Frequency Generation in EMG-to-Speech Conversion Using a Quantization Approach

Diener, Lorenz; Umesh, T. G.; Schultz, Tanja

doi:10.1109/asru46091.2019.9003804

Cited by 10 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Silent Computational Paralinguistics (SCP) reveal paralinguistics for situations when audible acoustic signals are not available or advisable, e. g., due to privacy concerns or disturbance of others, adverse noise conditions, or speech pathologies. While SSIs have previously addressed Automatic Speech Recognition, e. g., from video, EMG, or ultrasound [5], or examined how to synthesize silent to audible speech, e. g., for laryngectomy patients [6,7,8], research on privacy for paralinguistic analysis has focused mostly on whispered speech [9,10,11]. Some research has explored EMG for emotion recognition [12], and facial expressions to enhance human-computer interaction [13] or human-robot interaction [14].…”

Section: Introductionmentioning

confidence: 99%

Towards Silent Paralinguistics: Deriving Speaking Mode and Speaker ID from Electromyographic Signals

et al. 2020

Self Cite

View full text Add to dashboard Cite

Silent Computational Paralinguistics (SCP) -the assessment of speaker states and traits from non-audibly spoken communication -has rarely been targeted in the rich body of either Computational Paralinguistics or Silent Speech Processing. Here, we provide first steps towards this challenging but potentially highly rewarding endeavour: Paralinguistics can enrich spoken language interfaces, while Silent Speech Processing enables confidential and unobtrusive spoken communication for everybody, including mute speakers. We approach SCP by using speech-related biosignals stemming from facial muscle activities captured by surface electromyography (EMG). To demonstrate the feasibility of SCP, we select one speaker trait (speaker identity) and one speaker state (speaking mode). We introduce two promising strategies for SCP: (1) deriving paralinguistic speaker information directly from EMG of silently produced speech versus (2) first converting EMG into an audible speech signal followed by conventional computational paralinguistic methods. We compare traditional feature extraction and decision making approaches to more recent deep representation and transfer learning by convolutional and recurrent neural networks, using openly available EMG data. We find that paralinguistics can be assessed not only from acoustic speech but also from silent speech captured by EMG.

show abstract

Section: Introductionmentioning

confidence: 99%

Towards Silent Paralinguistics: Deriving Speaking Mode and Speaker ID from Electromyographic Signals

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, Recurrent Neural Networks (RNNs) have been used to predict natural F 0 patterns for alaryngeal voices based on conventional spectral features in [28] and [8]. Similarly, Diener et al in [29] developed a prediction system based on electromyographic signals.…”

Section: B Vc-based Statistical F 0 Prediction and Voicing State Controlmentioning

confidence: 99%

“…In contrast to regression modeling where target F 0 values form a continuous output variable, in classification predictive modeling, output variables are discrete classes or levels. To put it differently, classification modeling turns the task of predicting a real value for F 0 into predicting the most-probable quantization level where the F 0 falls onto [29]. Therefore, by solving a multi-level classification problem, the mapping function from input features to discretized target F0 patterns can be approximated.…”

Section: ) Classification Predictive Modelingmentioning

confidence: 99%

“…This feature-learning capability enables DL models to capture information on complex macro-and micro-structures and learn the underlying mappings between these structures and target variables, resulting in significant speedups compared to conventional methods. DL-based techniques have demonstrated success in alaryngeal speech enhancement [8], [26]- [29], notably in mapping EL speech spectral features into natural F 0 patterns using recurrent neural networks (RNNs) [8], [28]. Furthermore, sequence-to-sequence (seq2seq) modeling for EL to normal speech (EL2SP) conversion has been proposed [30], [31] using text-to-speech (TTS) pretraining [32] and the attention-based encoder-decoder framework [33].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Investigation of Fundamental Frequency Pattern Prediction for Japanese Electrolaryngeal Speech Enhancement Based on Frame-Wise Phoneme Representations

Eshghi,

Toda

2024

IEEE Access

View full text Add to dashboard Cite

Total laryngectomy (TL) stands as a well-established treatment for advanced laryngeal malignancies, entailing the complete removal of the larynx. Speech rehabilitation following TL is crucial for improving the quality of life (QoL) and facilitating social reintegration. Electrolaryngeal (EL) speech, a widely-used voice restoration technique utilizing external excitation signals, often produces artificial and monotonous sound quality despite enabling patients to form lengthy sentences. Efforts to enhance EL speech include the application of statistical voice conversion (VC) and neural approaches to speech enhancement. These approaches typically aim to map spectral features into acoustic characteristics, including the fundamental frequency (F 0 ). However, challenges arise due to substantial discrepancies and pattern changes between extracted features for EL and normal speech, compounded by limited clinical training data. To address this issue, we explored F 0 pattern prediction based on frame-wise phoneme information using bidirectional long short-term memory (BiLSTM) recurrent neural networks. Beyond direct predictions based on phoneme labels, we expanded our analysis to include real-valued phoneme embeddings, and conducted predictions for clustered embeddings representing lower-dimensional input representations. Our findings demonstrate that both regression and classification predictive modeling can map frame-wise phoneme information into natural F 0 patterns. Additionally, phoneme labels can be considered as shared features between EL and normal speech, allowing for improved prediction accuracies by incorporating phoneme information from normal speech into the training sets for EL speech. Furthermore, by learning of phoneme embeddings and creating input features for F 0 prediction based on the clustering of these embeddings, accurate F 0 patterns can be predicted, and the challenge of finding a strategy to reduce the dimensionality of the input features can be effectively alleviated.INDEX TERMS Electrolaryngeal speech, fundamental frequency prediction, phoneme labels, phoneme embeddings, speech enhancement.Following TL, the pharynx is decoupled from the trachea, and inhalation and exhalation occur through an opening in

show abstract

“…Prosody is mainly conditioned by the airflow and the vibration of the vocal folds, which in the case of laryngectomised patients is not possible to recover. As a result, most direct synthesis techniques generating a voice from sensed articulatory movements can, at best, recover a monotonous voice with limited pitch variations [101], [281], [282]. The use of complementary information capable of restoring prosodic features is thus an important area for future research.…”

Section: A Improved Sensing Techniquesmentioning

confidence: 99%

Silent Speech Interfaces for Speech Restoration: A Review

et al. 2020

View full text Add to dashboard Cite

This review summarises the status of silent speech interface (SSI) research. SSIs rely on non-acoustic biosignals generated by the human body during speech production to enable communication whenever normal verbal communication is not possible or not desirable. In this review, we focus on the first case and present latest SSI research aimed at providing new alternative and augmentative communication methods for persons with severe speech disorders. SSIs can employ a variety of biosignals to enable silent communication, such as electrophysiological recordings of neural activity, electromyographic (EMG) recordings of vocal tract movements or the direct tracking of articulator movements using imaging techniques. Depending on the disorder, some sensing techniques may be better suited than others to capture speech-related information. For instance, EMG and imaging techniques are well suited for laryngectomised patients, whose vocal tract remains almost intact but are unable to speak after the removal of the vocal folds, but fail for severely paralysed individuals. From the biosignals, SSIs decode the intended message, using automatic speech recognition or speech synthesis algorithms. Despite considerable advances in recent years, most present-day SSIs have only been validated in laboratory settings for healthy users. Thus, as discussed in this paper, a number of challenges remain to be addressed in future research before SSIs can be promoted to real-world applications. If these issues can be addressed successfully, future SSIs will improve the lives of persons with severe speech impairments by restoring their communication capabilities.

show abstract

Improving Fundamental Frequency Generation in EMG-to-Speech Conversion Using a Quantization Approach

Cited by 10 publications

References 12 publications

Towards Silent Paralinguistics: Deriving Speaking Mode and Speaker ID from Electromyographic Signals

Towards Silent Paralinguistics: Deriving Speaking Mode and Speaker ID from Electromyographic Signals

An Investigation of Fundamental Frequency Pattern Prediction for Japanese Electrolaryngeal Speech Enhancement Based on Frame-Wise Phoneme Representations

Silent Speech Interfaces for Speech Restoration: A Review

Contact Info

Product

Resources

About