Speaker Adaptation for Lip-Reading Using Visual Identity Vectors

Abstract

Visual speech recognition or lipreading suffers from high word error rate (WER) as lipreading is based solely on articulators that are visible to the camera. Recent works mitigated this problem using complex architectures of deep neural networks. Ivector based speaker adaptation is a well known technique in ASR systems used to reduce WER on unseen speakers. In this work, we explore speaker adaptation of lipreading models using latent identity vectors (visual i-vectors) obtained by factor analysis on visual features. In order to estimate the visual i-vectors, we employ two ways to collect sufficient statistics: first using GMM based universal background model (UBM) and second using RNN-HMM based UBM. The speaker-specific visual i-vector is given as an additional input to the hidden layers of the lipreading model during train and test phases. On GRID corpus, use of visual i-vectors results in 15% and 10% relative improvements over current state of the art lipreading architectures on unseen speakers using RNN-HMM and GMM based methods respectively. Furthermore, we explore the variation of WER with dimension of visual i-vectors, and with the amount of unseen speaker data required for visual i-vector estimation. We also report the results on Korean visual corpus that we created.

Comparing Speaker Adaptation Methods for Visual Speech Recognition for Continuous Spanish

“…In fact, it has been proven that each person produces speech in a unique way [36], a finding that supports the idea that visual speech features are highly sensitive to the identity of the speaker [29]. However, although a wide range of works have studied the speaker adaptation of end-to-end systems in the field of ASR [37][38][39][40], only a few works in this regard have addressed VSR [41,42]. Although this speaker-dependent approach makes for a less demanding task, it should not be forgotten that speaker-adapted VSR systems could be helpful in a non-invasive and inconspicuous way for people who suffer from communication difficulties [15][16][17].…”

“…Although this research describes approaches that could be adopted to any speech modality, it is noteworthy that few works have explicitly focused on speaker adaptation for VSR systems. Kandala et al [41] defined an architecture based on the CTC paradigm [23] where, after computing visual speech features, a speaker-specific identity vector was integrated as an additional input to the decoder. Fernandez-Lopez et al [42] approached the problem indirectly, studying how to adapt the visual front end of an audiovisual recognition system.…”

Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish

“…Albeit this research describes approaches that could be adopted to any speech modality, it is noteworthy that few works have focused specifically on speaker adaptation for VSR systems. Kandala et al [16] defined an architecture based on the Connectionist Temporal Classification (CTC) paradigm [26] where, once visual speech features were computed, a speakerspecific identity vector was integrated as an additional input to the decoder. Moreover, Fernandez-Lopez et al [17] approached the problem indirectly, since it was studied how to adapt the visual front-end of an audio-visual recognition system.…”

“…As detailed in Section 2, there is a wide range of works which have studied the speaker adaptation of end-toend systems in the field of Acoustic Speech Recognition (ASR) [12,13,14,15]. On the contrary, few works in this regard have been addressed in VSR [16,17]. Albeit this speaker-dependent approach means facing a less demanding task, it should not be forgotten that speaker-adapted VSR systems could be helpful, in a non-invasive and inconspicuous way, for people who suffer from communication difficulties [18,19].…”