How Phonotactics Affect Multilingual and Zero-Shot ASR Performance

Abstract

The idea of combining multiple languages' recordings to train a single automatic speech recognition (ASR) model brings the promise of the emergence of universal speech representation. Recently, a Transformer encoder-decoder model has been shown to leverage multilingual data well in IPA transcriptions of languages presented during training. However, the representations it learned were not successful in zero-shot transfer to unseen languages. Because that model lacks an explicit factorization of the acoustic model (AM) and language model (LM), it is unclear to what degree the performance suffered from differences in pronunciation or the mismatch in phonotactics. To gain more insight into the factors limiting zero-shot ASR transfer, we replace the encoder-decoder with a hybrid ASR system consisting of a separate AM and LM. Then, we perform an extensive evaluation of monolingual, multilingual, and crosslingual (zeroshot) acoustic and language models on a set of 13 phonetically diverse languages. We show that the gain from modeling crosslingual phonotactics is limited, and imposing a too strong model can hurt the zero-shot transfer. Furthermore, we find that a multilingual LM hurts a multilingual ASR system's performance, and retaining only the target language's phonotactic data in LM training is preferable.

Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition

“…Interestingly, we see that the hybrid system with a weak phonotactic model performs best (68.6%), and the one with a strong phonotactic model performs worst (66.7%). This finding is consistent with our previous hypothesis that learning non-target languages' phonotactics is harmful for a zero-shot ASR system [22] [RQ3]. If the phonotactic model had perfect knowledge of the phonotactics of the target language, the F1-score raises to 81.8%, which is mostly due to a perfect precision score of 100%.…”

“…So, despite lack of explicit constraint on the phonotactic information, we see an increase in F1 scores for the E2E ASR multi models. This is consistent with the hypothesis of [22] that the encoder-decoder transformer systems are learning a representation of the target languages' phonotactics [RQ3]. Finally, the best result of 90.6% F1score is obtained by a fully monolingual hybrid system with a word-level trigram LM, likely due to its stronger phonotactic constraint imposed by the presence of a decoding graph.…”

“…Ultimately, we want to understand the effect of phonotactics on phonetic inventory discovery. For this reason, we include and assert our previous finding from [22] here. The summary in Table 2 clearly shows that the tg-cross LM has the worst scores all the systems in this study.…”

“…To the best of our knowledge, the presented study is the first to address the problem of automatic phonetic inventory discovery using IPA symbols with an application of ASR systems. It concludes a series of works that began with the exploratory analysis of the universality of phonetic representations in E2E ASR [21], and then moved onto studying the effect of phonotactic models strength on the performance of crosslingual (zero-shot) hybrid ASR [22]. These works have laid the ground for addressing our ultimate goal -i.e., phonetic inventory discovery.…”

“…They will serve as oracles for evaluating phonetic inventory discovery. Hence, our ASR experiments are divided into three scenarios (mimicking our previous setups [21,22]; see also Figure 4):…”