Disfluency Detection with Unlabeled Data and Small BERT Models

Rocholl, Johann C.; Zayats, Vicky; Walker, Daniel D.; Murad, Noah B.; Schneider, Aaron; Liebling, Daniel J.

doi:10.21437/interspeech.2021-351

Cited by 16 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to shorter word-level latency metrics presented in the results, the runtime latency of the BERTSV model is 80% lower than that of BERTBASE(Rocholl et al, 2021).…”

mentioning

confidence: 73%

“…We specifically use the version from the Linguistic Data Corpus's Treebank-3 (Marcus et al, 1999) distribution, which additionally contains disfluency annotations and a standard train/dev/test split (Charniak and Johnson, 2001). We follow Rocholl et al (2021), training our models to classify both the reparanda and interregna as disfluent for future removal in a final post-processed transcript.…”

Section: Methodsmentioning

confidence: 99%

“…We use small models for two reasons: 1) finetuning a model on all given prefixes of each training example is resource intensive, and 2) many streaming natural language understanding applications run entirely on mobile devices which precludes the use of large models. Previous work on small non-incremental BERT-based models used for disfluency detection (Rocholl et al, 2021) showed significant improvement in memory and latency without compromising task performance. The core BERT SV model is a distilled version of BERT BASE with smaller vocabulary and reduced hidden layer dimensions (Zhao et al, 2021).…”

Section: Baselinesmentioning

confidence: 99%

See 2 more Smart Citations

Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection

Chen¹,

Zayats²,

Walker³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

In modern interactive speech-based systems, speech is consumed and transcribed incrementally prior to having disfluencies removed. This post-processing step is crucial for producing clean transcripts and high performance on downstream tasks (e.g. machine translation). However, most current state-of-theart NLP models such as the Transformer operate non-incrementally, potentially causing unacceptable delays. We propose a streaming BERT-based sequence tagging model that, combined with a novel training objective, is capable of detecting disfluencies in real-time while balancing accuracy and latency. This is accomplished by training the model to decide whether to immediately output a prediction for the current input or to wait for further context. Essentially, the model learns to dynamically size its lookahead window. Our results demonstrate that our model produces comparably accurate predictions and does so sooner than our baselines, with lower flicker. Furthermore, the model attains state-of-the-art latency and stability scores when compared with recent work on incremental disfluency detection.

show abstract

“…In addition to shorter word-level latency metrics presented in the results, the runtime latency of the BERTSV model is 80% lower than that of BERTBASE(Rocholl et al, 2021).…”

mentioning

confidence: 73%

Section: Methodsmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

See 1 more Smart Citation

Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection

Chen¹,

Zayats²,

Walker³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

show abstract

“…Disfluency detection is a widely studied area of research, with the most successful approaches leveraging BERT transformer models to achieve high accuracy (e.g. Bach and Huang, 2019;Jamshid Lou and Johnson, 2020;Rocholl et al, 2021). These models operate non-incrementally using whole sentences as inputs, often with a view to remove the disfluencies from transcripts all together.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

2022

View full text Add to dashboard Cite

As scalable learning technologies become ubiquitous, it generates a large amount of student data, which can be used with machine learning and NLP to develop new instructional technologies, such as personalized practice schedules and adaptive lessons. Additionally, machine learning and NLP are uniquely poised to solve the problems inherent in scaling language instruction to a large number of languages and courses. In this talk, I will describe several projects illustrating these two uses of ML and NLP in language learning at scale at Duolingo -the world's largest language education platform with over 100 courses and around 40 million monthly active learners.

show abstract

“…For the filled pause, we simply relabel the silence following predefined filler words as <pause> . Finally, we exploit the disfluency detector [18] based on small vocabulary BERT to identify reparandum and interregnum (e.g. "you know", "well", "I mean") of disfluencies.…”

Section: Training Data Annotationmentioning

confidence: 99%

Turn-Taking Prediction for Natural Conversational Speech

Chang¹,

Li²,

Sainath³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

While a streaming voice assistant system has been used in many applications, this system typically focuses on unnatural, one-shot interactions assuming input from a single voice query without hesitation or disfluency. However, a common conversational utterance often involves multiple queries with turn-taking, in addition to disfluencies. These disfluencies include pausing to think, hesitations, word lengthening, filled pauses and repeated phrases. This makes doing speech recognition with conversational speech, including one with multiple queries, a challenging task. To better model the conversational interaction, it is critical to discriminate disfluencies and end of query in order to allow the user to hold the floor for disfluencies while having the system respond as quickly as possible when the user has finished speaking. In this paper, we present a turntaking predictor built on top of the end-to-end (E2E) speech recognizer. Our best system is obtained by jointly optimizing for ASR task and detecting when the user is paused to think or finished speaking. The proposed approach demonstrates over 97% recall rate and 85% precision rate on predicting true turn-taking with only 100 ms latency on a test set designed with 4 types of disfluencies inserted in conversational utterances.

show abstract

Disfluency Detection with Unlabeled Data and Small BERT Models

Cited by 16 publications

References 0 publications

Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection

Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

Turn-Taking Prediction for Natural Conversational Speech

Contact Info

Product

Resources

About