Audiovisual speech recognition using multiscale nonlinear image decomposition

Abstract

There has recently been increasing interest in the idea of enhancing speech recognition by the use of visual information derived from the face of the talker. This paper demonstrates the use of nonlinear image decomposition, in the form of a 'sieve', applied to the task of visual speech recognition. Information derived from the mouth region is used in visual and audiovisual speech recognition of a database of the letters A-Z for four talkers. A scale histogram is generated directly from the grayscale pixels of a window containing the talkers mouth on a per frame basis. Results are presented for visual-only, audio-only and in a simple audiovisual case.

Recent advances in the automatic recognition of audiovisual speech

“…Such improvements have been typically demonstrated on databases of small duration, and, in most cases, limited to a very small number of speakers (mostly less than ten, and often singlesubject) and to small vocabulary tasks [18], [21]. Common tasks typically include recognition of non-sense words [22], [23], isolated words [19], [24][25][26][27][28][29][30], connected digits [31], [32], letters [31], or of closed-set sentences [33], mostly in English, but also in French [22], [34], [35], German [36], [37], and Japanese [38], among others. Recently however, significant improvements have also been demonstrated for large vocabulary continuous speech recognition (LVCSR) [39], as well as cases of speech degraded due to speech impairment [40] or Lombard effects [29].…”

“…1). The first scenario is primarily useful in benchmarking the performance of visual feature extraction algorithms, with visual-only ASR results typically reported on small-vocabulary tasks [24], [25], [28][29][30][31], [36], [40][41][42][43], [46], [59], [66], [78], [84][85][86][87][88][89][90][91][92]. Visual speech modeling is required in this process, its two central aspects being the choice of speech classes, that are assumed to generate the observed features, and the statistical modeling of this generation process.…”

“…Then, we seek MLLR adapted HMM parameters (see also (2) Closed form solutions for the unknown matrices exist, if the HMM covariances are diagonal [129]. Tulips1 I-4 digits 12 US [24], [46], [59], [78], [84], [85] M2VTS I-digits 37 FR [27], [114], [134] UIUC C-digits 100 US [32], [86] CUAVE I/C-digits 36 US [28], [87], [135] IBM C-digits 50 US [40] U.Karlsruhe C-letters 6 D [36], [41], [42], [88] U.LeMans I-letters 2 FR [34], [35], [93], [100] U.Sheffield I-letters 10 UK [25], [89] AT&T C-letters 49 US [31], [43], [90] UT.Austin I-500 words 1 US [95] AMP-CMU I-78 words 10 US [29], [30], [66], [91], [92] ATR I-words 1 JP [38], [123] AV-TIMIT 150-sent.…”

Automatic Visual Speech Recognition

“…The authors report the correct recognition close to 90%. However, based on the AVletters data corpus, Matthews et al (Matthews et al, 1996) reports only a 50% recognition rate. Li et al (Li et al, 1995) reports a perfect recognition 100% on the same task, but two years later in (Li et al, 1997) only 90% recognition.…”

“…In all cases, the first and probably the most important step in building a data corpus is to carefully state the targeted applications of the system that will be trained using the dataset. Some of the most cited data corpora for lip reading are: TULIPS1 (Movellan, 1995), AVletters (Matthews et al, 1996), AVOZES (Goecke & Millar, 2004), CUAVE (Patterson et al, 2002), DAVID (Chibelushi et al, 1996), ViaVoice (Neti et al, 2000), DUTAVSC (Wojdel et al, 2002), AVICAR (Lee et al, 2004), AT&T (Potamianos et al, 1997), CMU (Zhang et al, 2002), XM2VTSDB (Messer et al, 1999), M2VTS (Pigeon & Vandendorpe, 1997) and LIUM-AVS (Daubias & Deleglise, 2003). With the exception of M2VTS which is in French, XM2VTSDB which is in four languages and DUTAVSC which is in Dutch the rest are only in English (Table 1).…”