Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Dalmia, Siddharth; Yan, Brian; Raunak, Vikas; Metze, Florian; Watanabe, Shinji

doi:10.18653/v1/2021.naacl-main.151

Cited by 19 publications

(9 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the NAR side, they use CMLM and a CTC-based model as NAR decoders, denoted as Orthros-CMLM and Orthros-CTC, respectively. Such muti-decoder is also widely used for speech translation [141], [142], which is a two-pass decoding method that decomposes the overall task into two sub-tasks, i.e., ASR and machine translation. Inaguma et al [143] propose Fast-MD, where the hidden intermediates are generated in a non-autoregressive manner by a Mask-CTC model.…”

Section: Speech Translationmentioning

confidence: 99%

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Xiao¹,

Wu²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, auto-regressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as dialogue generation, text summarization, grammar error correction, semantic parsing, speech synthesis, and automatic speech recognition. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, dynamic length prediction, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications.

show abstract

Section: Speech Translationmentioning

confidence: 99%

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Xiao¹,

Wu²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The performance of the downstream NLP task is directly affected by the recognition errors [31]. Previous studies improved the robustness of the NLP back-end to ASR errors using various auxiliary information sources from the ASR system, e.g., probabilities, recognition hypotheses, and hidden states [4,6,8,10,11,[32][33][34][35][36][37].…”

Section: Conventional Interconnection Of Asr and Ts Systemsmentioning

confidence: 99%

Attention-based Multi-hypothesis Fusion for Speech Summarization

Kano¹,

Ogawa²,

Delcroix³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Speech summarization, which generates a text summary from speech, can be achieved by combining automatic speech recognition (ASR) and text summarization (TS). With this cascade approach, we can exploit state-of-the-art models and large training datasets for both subtasks, i.e., Transformer for ASR and Bidirectional Encoder Representations from Transformers (BERT) for TS. However, ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary. We investigate several schemes to combine ASR hypotheses. First, we propose using the sum of sub-word embedding vectors weighted by their posterior values provided by an ASR system as an input to a BERT-based TS system. Then, we introduce a more general scheme that uses an attention-based fusion module added to a pre-trained BERT module to align and combine several ASR hypotheses. Finally, we perform speech summarization experiments on the How2 dataset and a newly assembled TED-based dataset that we will release with this paper 1 . These experiments show that retraining the BERT-based TS system with these schemes can improve summarization performance and that the attention-based fusion module is particularly effective.

show abstract

“…This year we focused on (1) sequence-level knowledge distillation (SeqKD) (Kim and Rush, 2016), (2) Conformer encoder (Gulati et al, 2020), (3) Multi-Decoder architecture (Dalmia et al, 2021), (4) model ensembling, and (5) better segmentation with a neural network-based voice activity (VAD) system (Bredin et al, 2020) and a novel algorithm to merge multiple short segments for long context modeling. Our primary focus was E2E models, although we also compared them with cascade systems with our best effort.…”

Section: Introductionmentioning

confidence: 99%

ESPnet-ST IWSLT 2021 Offline Speech Translation System

Inaguma¹,

Yan²,

Dalmia³

et al. 2021

Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)

Self Cite

View full text Add to dashboard Cite

This paper describes the ESPnet-ST group's IWSLT 2021 submission in the offline speech translation track. This year we made various efforts on training data, architecture, and audio segmentation. On the data side, we investigated sequence-level knowledge distillation (SeqKD) for end-to-end (E2E) speech translation. Specifically, we used multi-referenced SeqKD from multiple teachers trained on different amounts of bitext. On the architecture side, we adopted the Conformer encoder and the Multi-Decoder architecture, which equips dedicated decoders for speech recognition and translation tasks in a unified encoder-decoder model and enables search in both source and target language spaces during inference. We also significantly improved audio segmentation by using the pyannote.audio toolkit and merging multiple short segments for long context modeling. Experimental evaluations showed that each of them contributed to large improvements in translation performance. Our best E2E system combined all the above techniques with model ensembling and achieved 31.4 BLEU on the 2-ref of tst2021 and 21.2 BLEU and 19.3 BLEU on the two single references of tst2021. * *Equal contribution 1 https://sites.google.com/ view/iwslt-evaluation-2019/ speech-translation/off-limit-ted-talks 2 https://ict.fbk.eu/ must-c-release-v2-0/

show abstract

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Cited by 19 publications

References 58 publications

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Attention-based Multi-hypothesis Fusion for Speech Summarization

ESPnet-ST IWSLT 2021 Offline Speech Translation System

Contact Info

Product

Resources

About