Detecting Multiple Speech Disfluencies Using a Deep Residual Network with Bidirectional Long Short-Term Memory

Abstract

Stuttering is a speech impediment affecting tens of millions of people on an everyday basis. Even with its commonality, there is minimal data and research on the identification and classification of stuttered speech. This paper tackles the problem of detection and classification of different forms of stutter. As opposed to most existing works that identify stutters with language models, our work proposes a model that relies solely on acoustic features, allowing for identification of several variations of stutter disfluencies without the need for speech recognition. Our model uses a deep residual network and bidirectional long short-term memory layers to classify different types of stutters and achieves an average miss rate of 10.03%, outperforming the state-of-the-art by almost 27%.

Improved Robustness to Disfluencies in Rnn-Transducer Based Speech Recognition

“…In most of the cases the overall system maintains its modular setting and therefore requires not only separate optimization criteria for different components but also in some cases hand-labeled features for the annotation of the disfluencies to train the language model. A recent work brings the focus on the acoustic side and does not take into consideration any language-dependent information [12]. To the best of the authors' knowledge, with the exception of a work on personalized ASR for dysarthric speech [13], none of the published papers are aimed to improve speech recognition accuracy of an E2E ASR system by dealing with disfluencies without solving disfluency detection task itself.…”

SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter

“…Our approach takes an audio clip, extracts acoustic features per-frame, applies a temporal model, and outputs a single set of clip-level dysfluency labels. We investigated baselines that are inspired by the dysfluency model in [10] and alternative input features, model architectures, and loss functions.…”

“…A major bottleneck in this area is that dysfluency datasets tend to be small and have few or inconsistent annotations not inherently designed for work on speech recognition tasks. Kourkounakis et al [10] used 800 speech clips (53 minutes) with custom annotations to detect dysfluencies from 25 children who stutter using the UCLASS dataset [11]. Riad et al [12] performed a similar task using 1429 utterances from 22 adults who stutter with the recent FluencyBank [13] dataset.…”

“…The focus of this paper is on detection of five stuttering event types: Blocks, Prolongations, Sound Repetitions, Word/Phrase Repetitions, and Interjections. Existing work has explored this problem using traditional signal processing techniques [15,16,17], language modeling (LM) [12,18,19,20,21], and acoustic modeling (AM) [21,10]. Each approach has be shown to be effective 1.…”

“…at identifying one or two event types typically on data from a small number of users. Prolongations, or extended sounds, have been detected using short-window autocorrelations [16] and low-level acoustic models [10]. Word/phrase repetitions, if they are well articulated, are easily detected using LMbased approaches [19], with the caveat that single-syllable words such as in the phrase "I-I-I am" will often be smoothed into "I am" due to the underlying acoustic model and phrases like "I am [am]" may be pruned because the LM has never seen the word "am" repeated before.…”