Recent Progress in the CUHK Dysarthric Speech Recognition System

Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition

“…In contrast, related previous works either used: a) synthesized normal speech acoustic-articulatory features trained A2A inversion models before being applied to dysarthric speech [23], while the large mismatch between normal and impaired speech encountered during inversion model training and articulatory feature generation stages was not taken into account; or b) only considered the cross-domain or cross-corpus A2A inversion [25] while the quality of generated articulatory features was not assessed using the back-end disordered speech recognition systems. In addition, the lowest published WER of 24.82% on the benchmark UASpeech task in comparison against recent researches [8][9][10][11][12][13][37][38][39] was obtained using the proposed cross-domain acoustic-to-articulatory inversion approach.…”

“…Due to the acoustic domain mismatch, a direct cross-domain application of the A2A inversion model (described in Section 3.1) trained on the TORGO acoustic-articulatory parallel data to the UASpeech acoustic data is problematic, as was shown in the previous research on cross-domain audio-visual inversion [12]. To this end, the large acoustic domain mismatch between the two data sets can be minimized using multi-level adaptive networks (MLAN) [12,39,43] before A2A inversion can be performed. An example MLAN model is shown in the left part of Figure 2 (circled in red dotted line).…”

“…9 vs. Sys. 7) before and after data augmentation and LHUC-SAT based speaker adaptation applied to a state-of-the-art hybird DNN system [39], a CTC end-to-end system (Sys. 13 vs. Sys.…”

“…9), although with increased system complexity. The lowest WER of 24.82% was obtained by further incorporating visual features [39] (Sys. 11).…”

“…"DA" stands for data augmentation. systems WER% Sheffield-2013 Cross domain augmentation [8] 37.50 Sheffield-2015 Speaker adaptive training [9] 34.80 CUHK-2018 DNN System Combination [10] 30.60 Sheffield-2020 Fine-tuning CNN-TDNN speaker adaptation [11] 30.76 CUHK-2020 Cross-domain AVSR [12] 26.84 CUHK-2020 DNN + DA + LHUC SAT [37] 26.37 CUHK-2021 DNN + GAN based DA + LHUC SAT [38] 25.89 CUHK-2021 NAS DNN + DA + LHUC SAT + AV fusion [39] 25.21 DNN + DA + LHUC SAT + AAV fusion (ours) 24.82…”