Automatic Speech Disfluency Detection Using wav2vec2.0 for Different Languages with Variable Lengths

Abstract

Speech is critical for interpersonal communication, but not everyone has fluent communication skills. Speech disfluency, including stuttering and interruptions, affects not only emotional expression but also clarity of expression for people who stutter. Existing methods for detecting speech disfluency rely heavily on annotated data, which can be costly. Additionally, these methods have not considered the issue of variable-length disfluent speech, which limits the scalability of detection methods. To address these limitations, this paper proposes an automated method for detecting speech disfluency that can improve communication skills for individuals and assist therapists in tracking the progress of stuttering patients. The proposed method focuses on detecting four types of disfluency features using single-task detection and utilizes embeddings from the pre-trained wav2vec2.0 model, as well as convolutional neural network (CNN) and Transformer models for feature extraction. The model’s scalability is improved by considering the issue of variable-length disfluent speech and modifying the model based on the entropy invariance of attention mechanisms. The proposed automated method for detecting speech disfluency has the potential to assist individuals in overcoming speech disfluency, improve their communication skills, and aid therapists in tracking the progress of stuttering patients. Additionally, the model’s scalability across languages and lengths enhances its practical applicability. The experiments demonstrate that the model outperforms baseline models in both English and Chinese datasets, proving its universality and scalability in real-world applications.

Classification of Speech Arrests and Speech Impairments during Awake Craniotomy: a multi-databases analysis

“…The most widely studied speech disorder is stuttering [14][15][16][17][18][19]. The developed approaches to detect stuttering events usually rely on two datasets, SEP-28k [20] and FluencyBank [21] which are used to train, validate and compare the proposed approaches.…”

“…Several approaches have been proposed, for instance, MFCC features have been used to train a neural network comprising an LSTM layer [15]. However, most of the approaches today [14,[17][18][19] use Wav2Vec2 [1]. Wav2vec2 has the advantage of taking a raw audio waveform as input, thus avoiding a laborious step of parameters selection for the feature extraction, and to learn features with a reasonable amount of data as it has many pre-trained models in different languages and for different tasks.…”