An Effective Contextual Language Modeling Framework for Speech Summarization with Augmented Features

Weng, Shi-Yan; Lo, Tien-Hong; Chen, Berlin

doi:10.23919/eusipco47968.2020.9287432

Cited by 6 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance of the downstream NLP task is directly affected by the recognition errors [31]. Previous studies improved the robustness of the NLP back-end to ASR errors using various auxiliary information sources from the ASR system, e.g., probabilities, recognition hypotheses, and hidden states [4,6,8,10,11,[32][33][34][35][36][37].…”

Section: Conventional Interconnection Of Asr and Ts Systemsmentioning

confidence: 99%

“…For speech summarization, Weng et al proposed including a confidence embedding in the input of BERTSum [29] to achieve robust speech summarization [4]. They modified the input embedding vectors of BERTSum to be the sum of the sub-word embedding vector and a confidence embedding.…”

Section: Conventional Interconnection Of Asr and Ts Systemsmentioning

confidence: 99%

“…However, while generating the summary, a cascade connection discards speech-specific information [2,3], which has the potential to enrich the summary, such as intonation when generating the summary. Moreover, the TS system receives input text containing ASR errors that affect the performance of the speech summarization system [4]. This study focuses on the latter problem.…”

Section: Introductionmentioning

confidence: 99%

“…For speech summarization, some studies built systems that are robust to ASR errors [11][12][13]. Weng et al [4] achieved robust speech summarization by adding confidence scores associated with each recognized word to the input of a BERT-based TS system. The confidence scores help the TS system to ignore unreliable words in the ASR output.…”

Section: Introductionmentioning

confidence: 99%

“…Posterior fusion is similar to the confidence-based approach [4] in the sense that both approaches simply exploit the confidence of individual words or tokens within the 1-best hypothesis at token level and do not explicitly model multiple hypotheses at sequence level. In contrast, the attention fusion explicitly models multiple token sequence hypotheses by using two attention steps.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Attention-based Multi-hypothesis Fusion for Speech Summarization

Kano¹,

Ogawa²,

Delcroix³

et al. 2021

Preprint

View full text Add to dashboard Cite

Speech summarization, which generates a text summary from speech, can be achieved by combining automatic speech recognition (ASR) and text summarization (TS). With this cascade approach, we can exploit state-of-the-art models and large training datasets for both subtasks, i.e., Transformer for ASR and Bidirectional Encoder Representations from Transformers (BERT) for TS. However, ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary. We investigate several schemes to combine ASR hypotheses. First, we propose using the sum of sub-word embedding vectors weighted by their posterior values provided by an ASR system as an input to a BERT-based TS system. Then, we introduce a more general scheme that uses an attention-based fusion module added to a pre-trained BERT module to align and combine several ASR hypotheses. Finally, we perform speech summarization experiments on the How2 dataset and a newly assembled TED-based dataset that we will release with this paper 1 . These experiments show that retraining the BERT-based TS system with these schemes can improve summarization performance and that the attention-based fusion module is particularly effective.

show abstract