ESPnet-SE: End-To-End Speech Enhancement and Separation Toolkit Designed for ASR Integration

Li, Chenda; Jing, Shi; Zhang, Wangyou; Subramanian, Arvind; Chang, Xuankai; Kamo, Naoyuki; Hira, Moto; Hayashi, Tomoki; Boeddeker, Christoph; Chen, Zhuo; Watanabe, Shinji

doi:10.1109/slt48900.2021.9383615

Cited by 70 publications

(28 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the exception of GPT-3 and ERNIE 3.0, most PTMs in the literature can be downloaded from hubs such as PaddleHub, 5 HuggingFaceHub, 6 and Fairseq 7 (Ott et al 2019). Historically, many of these hubs started with models for natural language processing, though they are beginning to expand into other fields such as speech (SpeechBrain 8 (Ravanelli et al 2021) and ESPnet 9 (Watanabe et al 2018;Hayashi et al 2020;Inaguma et al 2020;Li et al 2021)) and vision (VIT 10 (Dosovitskiy et al 2021;Tolstikhin et al 2021;Steiner et al 2021;Chen, Hsieh, and Gong 2021)).…”

Section: Pre-training Fine-tuning and Inferencementioning

confidence: 99%

Emerging trends: A gentle introduction to fine-tuning

Church¹,

Chen

2021

Nat. Lang. Eng.

View full text Add to dashboard Cite

The previous Emerging Trends article (Church et al., 2021. Natural Language Engineering27(5), 631–645.) introduced deep nets to poets. Poets is an imperfect metaphor, intended as a gesture toward inclusion. The future for deep nets will benefit by reaching out to a broad audience of potential users, including people with little or no programming skills, and little interest in training models. That paper focused on inference, the use of pre-trained models, as is, without fine-tuning. The goal of this paper is to make fine-tuning more accessible to a broader audience. Since fine-tuning is more challenging than inference, the examples in this paper will require modest programming skills, as well as access to a GPU. Fine-tuning starts with a general purpose base (foundation) model and uses a small training set of labeled data to produce a model for a specific downstream application. There are many examples of fine-tuning in natural language processing (question answering (SQuAD) and GLUE benchmark), as well as vision and speech.

show abstract

Section: Pre-training Fine-tuning and Inferencementioning

confidence: 99%

Emerging trends: A gentle introduction to fine-tuning

Church¹,

Chen

2021

Nat. Lang. Eng.

View full text Add to dashboard Cite

show abstract

“…We adopt the 5-th channel (CH5) as the reference channel for both training and evaluation. For evaluating the performance of frontend models on the real data, we adopt an E2E ASR model pretrained on the CHiME-4 dataset, which was also used in Section 4.1 in [24]. For the joint training of frontend and backend, we optionally include an additional dataset from the Wall Street Journal (WSJ) corpus [25] for training, which consists of 37416 clean speech samples.…”

Section: Experiments 31 Experimental Setupmentioning

confidence: 99%

“…All our models are built based on the ESPnet toolkit [24,27]. The MC-Conv-TasNet model uses a Conv1D layer with 5 input channels and 256 output channels for the multi-channel encoder, with a kernel size of 20 and stride of 10.…”

Section: Experiments 31 Experimental Setupmentioning

confidence: 99%

“…Here we take the official result [16] as the ASR baseline, which uses a DNN-HMM acoustic model with language model rescoring. The speech enhancement baselines include the BeamformIt [32], neural beamformer (denoted as "BLSTM MVDR") from [24], and the timedomain filter-and-sum network (FaSNet) 2 [31]. The speech recognition performance is evaluated using the same pretrained E2E ASR model on CHiME-4 for models from No.…”

Section: Performance Evaluation On Simulation and Real Datamentioning

confidence: 99%

See 1 more Smart Citation

Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions

Zhang¹,

Shi²,

Li³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The deep learning based time-domain models, e.g. Conv-TasNet, have shown great potential in both single-channel and multi-channel speech enhancement. However, many experiments on the timedomain speech enhancement model are done in simulated conditions, and it is not well studied whether the good performance can generalize to real-world scenarios. In this paper, we aim to provide an insightful investigation of applying multi-channel Conv-TasNet based speech enhancement to both simulation and real data. Our preliminary experiments show a large performance gap between the two conditions in terms of the ASR performance. Several approaches are applied to close this gap, including the integration of multi-channel Conv-TasNet into the beamforming model with various strategies, and the joint training of speech enhancement and speech recognition models. Our experiments on the CHiME-4 corpus show that our proposed approaches can greatly reduce the speech recognition performance discrepancy between simulation and real data, while preserving the strong speech enhancement capability in the frontend.

show abstract

“…Thus, we can easily evaluate the E2E-ASR performance of pretrained SSLRs available in S3PRL using the current state-ofthe-art (SOTA) neural network models, such as Transformers [4,5] and Conformers [6]. We can also easily evalute the SSLRs in other downstream tasks, including speech translation (ST) [30] and speech enhancement (SE) [31]. It is also an interesting question in the air about the generalization ability of these SSLRs, given the fact that most of SSLRs were trained and tested mainly on LibriSpeech [7,32].…”

Section: Introductionmentioning

confidence: 99%

An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition

Chang¹,

Maekaku²,

Guo³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Self-supervised pretraining on speech data has achieved a lot of progress. High-fidelity representation of the speech signal is learned from a lot of untranscribed data and shows promising performance. Recently, there are several works focusing on evaluating the quality of self-supervised pretrained representations on various tasks without domain restriction, e.g. SUPERB. However, such evaluations do not provide a comprehensive comparison among many ASR benchmark corpora. In this paper, we focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR. Without any modification of the back-end model architectures or training strategy, some of the experiments with pretrained representations, e.g., WSJ, WSJ0-2mix with HuBERT, reach or outperform current state-of-the-art (SOTA) recognition performance. Moreover, we further explore more scenarios for whether the pretraining representations are effective, such as the cross-language or overlapped speech. The scripts, configuratons and the trained models have been released in ESPnet to let the community reproduce our experiments and improve them.

show abstract

ESPnet-SE: End-To-End Speech Enhancement and Separation Toolkit Designed for ASR Integration

Cited by 70 publications

References 38 publications

Emerging trends: A gentle introduction to fine-tuning

Emerging trends: A gentle introduction to fine-tuning

Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions

An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition

Contact Info

Product

Resources

About