Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition

Lohrenz, Timo; Li, Zhengyang; Fingscheidt, Tim

doi:10.48550/arxiv.2104.00120

Cited by 4 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fusion Fusion is another method for utilizing multiple candidates (Liu et al, 2021;Lohrenz et al, 2021). Specifically, Fusion uses a shared encoder to extract the representation of each candidate and The comparison results of FastCorrect 2 with the above two baselines are shown in Table 5.…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition

Leng

Tan

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Section: Comparison With Other Methodsmentioning

confidence: 99%

FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition

Leng

Tan

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For the multi-encoder transformer, researchers have focused on the different types of input values (sources) to be received and encoded and how to combine representations generated from the multi-encoder and use it in the decoder. Multi-encoder approaches have mainly been studied in relation to speech recognition [20], [21], [22], neural machine translation and automatic post-editing (APE), the task of correcting errors in machine-translated texts [10], [23], [24], [25], [26], [27]. In the existing multi-encoder method, two encoders are mainly used.…”

Section: Multi-encoder Transformermentioning

confidence: 99%

Multi-Encoder Transformer for Korean Abstractive Text Summarization

Shin

2023

IEEE Access

View full text Add to dashboard Cite

In this paper, we propose a Korean abstractive text summarization approach that uses a multi-encoder transformer. Recently, in many natural language processing (NLP) tasks, the use of the pre-trained language models (PLMs) for transfer learning has achieved remarkable performance. In particular, transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT) are used for pre-training and applied to downstream tasks, showing state-of-the-art performance including abstractive text summarization. However, existing text summarization models usually use one pre-trained model per model architecture, meaning that it becomes necessary to choose one PLM at a time. For PLMs applicable to Korean abstractive text summarization, there are publicly available BERT-based pretrained Korean models that offer different advantages such as Multilingual BERT, KoBERT, HanBERT, and KorBERT. We assume that if these PLMs could be leveraged simultaneously, better performance would be obtained. We propose a model that uses multiple encoders which are capable of leveraging multiple pre-trained models to create an abstractive summary. We evaluate our method using three benchmark Korean abstractive summarization datasets, each named Law (AI-Hub), News (AI-Hub), and News (NIKL) datasets. Experimental results show that the proposed multi-encoder model variations outperform single-encoder models. We find the empirically best summarization model by determining the optimal input combination when leveraging multiple PLMs with the multi-encoder method.INDEX TERMS Natural language processing, abstractive text summarization, bidirectional encoder representations from transformer, neural networks, natural language generation.

show abstract

“…Lohrenz et al. [ 24 ] proposed a multi-encoder learning and stream fusion for Transformer-based end-to-end automatic speech recognition. It is valuable to develop the new continuous HMR scheme based on Transformer-based encoder–decoder architecture.…”

Section: Related Workmentioning

confidence: 99%

Multi-Feature Transformer-Based Learning for Continuous Human Motion Recognition with High Similarity Using mmWave FMCW Radar

Chen

Cheng

et al. 2022

Sensors

View full text Add to dashboard Cite

Doppler-radar-based continuous human motion recognition recently has attracted extensive attention, which is a favorable choice for privacy and personal security. Existing results of continuous human motion recognition (CHMR) using mmWave FMCW Radar are not considered the continuous human motion with the high similarity problem. In this paper, we proposed a new CHMR algorithm with the consideration of the high similarity (HS) problem, called as CHMR-HS, by using the modified Transformer-based learning model. As far as we know, this is the first result in the literature to investigate the continuous HMR with the high similarity. To obtain the clear FMCW radar images, the background and target signals of the detected human are separated through the background denoising and the target extraction algorithms. To investigate the effects of the spectral-temporal multi-features with different dimensions, Doppler, range, and angle signatures are extracted as the 2D features and range-Doppler-time and range-angle-time signatures are extracted as the 3D features. The 2D/3D features are trained into the adjusted Transformer-encoder model to distinguish the difference of the high-similarity human motions. The conventional Transformer-decoder model is also re-designed to be Transformer-sequential-decoder model such that Transformer-sequential-decoder model can successfully recognize the continuous human motions with the high similarity. The experimental results show that the accuracy of our proposed CHMR-HS scheme are 95.2% and 94.5% if using 3D and 2D features, the simulation results also illustrates that our CHMR-HS scheme has advantages over existing CHMR schemes.

show abstract

Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition

Cited by 4 publications

References 37 publications

FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition

FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition

Multi-Encoder Transformer for Korean Abstractive Text Summarization

Multi-Feature Transformer-Based Learning for Continuous Human Motion Recognition with High Similarity Using mmWave FMCW Radar

Contact Info

Product

Resources

About