Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection

Abstract

In modern interactive speech-based systems, speech is consumed and transcribed incrementally prior to having disfluencies removed. This post-processing step is crucial for producing clean transcripts and high performance on downstream tasks (e.g. machine translation). However, most current state-of-theart NLP models such as the Transformer operate non-incrementally, potentially causing unacceptable delays. We propose a streaming BERT-based sequence tagging model that, combined with a novel training objective, is capable of detecting disfluencies in real-time while balancing accuracy and latency. This is accomplished by training the model to decide whether to immediately output a prediction for the current input or to wait for further context. Essentially, the model learns to dynamically size its lookahead window. Our results demonstrate that our model produces comparably accurate predictions and does so sooner than our baselines, with lower flicker. Furthermore, the model attains state-of-the-art latency and stability scores when compared with recent work on incremental disfluency detection.

MultiTurnCleanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup

“…Particularly, Table 4 illustrates the performance of Baseline and presented models on the two datasets. The STD module achieves cutting-edge performance (Chen et al, 2022) to detect singleturn disfluencies. Also, the MTD module outperforms the Baseline on detecting multi-turn discontinuities with our proposed MultiTurnCleanup dataset.…”

“…Compared with the existing disfluency detection task, which aims to detect disfluencies (e.g., selfrepairs, repetitions, restarts, and filled pauses) that commonly occur within single-turn utterances (Rocholl et al, 2021;Chen et al, 2022), the Multi-Turn Cleanup task requires identifying discontinuities both within a single turn and across multiple turns in the multi-party spoken conversational transcripts. To explicitly define the task and discontinuity taxonomy, we conducted an in-depth analysis of the Switchboard corpus 3 (Godfrey et al, 1992).…”

“…These phenomena act as noise that hampers human readability (Adda-Decker et al, 2003) and the performance of downstream tasks such as question answering (Gupta et al, 2021) or machine translation (Hassan et al, 2014) on transcripts of human spoken conversations. State-of-the-art disfluency detection methods (Yang et al, 2020;Jamshid Lou and Johnson, 2020) identify and remove disfluencies in order to improve the readability of spoken conversational transcripts (Wang et al, 2022;Chen et al, 2022). For instance, Figure 1(a) shows that disfluency detection methods can remove self-repairs of single turns.…”

The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling

“…The restart-incremental paradigm was investigated for Transformer-based sequence labelling by Madureira and Schlangen (2020) and Kahardipraja et al (2021); recently, adaptive policies were proposed to reduce the computational load (Kaushal et al, 2023;Kahardipraja et al, 2023). Rohanian and Hough (2021) and Chen et al (2022) explored adaptation strategies to use Transformers for incremental disfluency detection. In simultaneous translation, where policies are a central concept (Zheng et al, 2020a;Zhang et al, 2020), the restartincremental approach is in use and revisions are studied (Arivazhagan et al, 2020;Sen et al, 2023).…”