A new posterior based audio-visual integration method for robust speech recognition

Seymour, Rowan; Ji, Ming; Stewart, Darryl

doi:10.21437/interspeech.2005-375

Cited by 11 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we wish to carry out a complementary study in which we will compare the performance of a variety of different image transform-based feature types for speaker-independent visual speech recognition of isolated digits recorded in various noisy video conditions which may occur in real-world operating conditions. This work extends upon our previous research on the use of geometricbased features for audio-visual speech recognition subject to both audio and video corruptions [22].…”

Section: Introductionsupporting

confidence: 60%

Comparison of Image Transform-Based Features for Visual Speech Recognition in Clean and Corrupted Videos

Seymour¹,

Stewart²,

Ji³

2008

EURASIP Journal on Image and Video Processing

View full text Add to dashboard Cite

We present results of a study into the performance of a variety of different image transform-based feature types for speakerindependent visual speech recognition of isolated digits. This includes the first reported use of features extracted using a discrete curvelet transform. The study will show a comparison of some methods for selecting features of each feature type and show the relative benefits of both static and dynamic visual features. The performance of the features will be tested on both clean video data and also video data corrupted in a variety of ways to assess each feature type's robustness to potential real-world conditions. One of the test conditions involves a novel form of video corruption we call jitter which simulates camera and/or head movement during recording.

show abstract

Section: Introductionsupporting

confidence: 60%

Comparison of Image Transform-Based Features for Visual Speech Recognition in Clean and Corrupted Videos

Seymour¹,

Stewart²,

Ji³

2008

EURASIP Journal on Image and Video Processing

View full text Add to dashboard Cite

show abstract

“…In contrast to many other strategies, such as [ 10 , 33 , 34 ], reliability-based stream integration does not suffer from wide disparities in audio and video model performance. This is greatly beneficial to our case as we wish to design a system that least avoids any performance degradation due to the inclusion of multiple streams and that ideally profits from the visual modality under all, even under clean, acoustic conditions.…”

Section: Fusion Models Furthermore Baselinesmentioning

confidence: 99%

Reliability-Based Large-Vocabulary Audio-Visual Speech Recognition

Zeiler

Kolossa

2022

Sensors

View full text Add to dashboard Cite

Audio-visual speech recognition (AVSR) can significantly improve performance over audio-only recognition for small or medium vocabularies. However, current AVSR, whether hybrid or end-to-end (E2E), still does not appear to make optimal use of this secondary information stream as the performance is still clearly diminished in noisy conditions for large-vocabulary systems. We, therefore, propose a new fusion architecture—the decision fusion net (DFN). A broad range of time-variant reliability measures are used as an auxiliary input to improve performance. The DFN is used in both hybrid and E2E models. Our experiments on two large-vocabulary datasets, the Lip Reading Sentences 2 and 3 (LRS2 and LRS3) corpora, show highly significant improvements in performance over previous AVSR systems for large-vocabulary datasets. The hybrid model with the proposed DFN integration component even outperforms oracle dynamic stream-weighting, which is considered to be the theoretical upper bound for conventional dynamic stream-weighting approaches. Compared to the hybrid audio-only model, the proposed DFN achieves a relative word-error-rate reduction of 51% on average, while the E2E-DFN model, with its more competitive audio-only baseline system, achieves a relative word error rate reduction of 43%, both showing the efficacy of our proposed fusion architecture.

show abstract

“…In order to directly compare these values, which may be on different scales, we can normalize by converting to posterior probabilities. One method of selecting the optimal stream probability is the maximum stream posterior (MSP) method [23] that is expressed formally as follows:…”

Section: Maximum Weighted Stream Posterior Modelmentioning

confidence: 99%

Robust Audio-Visual Speech Recognition Under Noisy Audio-Video Conditions

Stewart

Seymour

Pass³

et al. 2014

IEEE Trans. Cybern.

Self Cite

View full text Add to dashboard Cite

This paper presents the maximum weighted stream posterior (MWSP) model as a robust and efficient stream integration method for audio-visual speech recognition in environments, where the audio or video streams may be subjected to unknown and time-varying corruption. A significant advantage of MWSP is that it does not require any specific measurements of the signal in either stream to calculate appropriate stream weights during recognition, and as such it is modality-independent. This also means that MWSP complements and can be used alongside many of the other approaches that have been proposed in the literature for this problem. For evaluation we used the large XM2VTS database for speaker-independent audio-visual speech recognition. The extensive tests include both clean and corrupted utterances with corruption added in either/both the video and audio streams using a variety of types (e.g., MPEG-4 video compression) and levels of noise. The experiments show that this approach gives excellent performance in comparison to another well-known dynamic stream weighting approach and also compared to any fixed-weighted integration approach in both clean conditions or when noise is added to either stream. Furthermore, our experiments show that the MWSP approach dynamically selects suitable integration weights on a frame-by-frame basis according to the level of noise in the streams and also according to the naturally fluctuating relative reliability of the modalities even in clean conditions. The MWSP approach is shown to maintain robust recognition performance in all tested conditions, while requiring no prior knowledge about the type or level of noise.

show abstract

A new posterior based audio-visual integration method for robust speech recognition

Cited by 11 publications

References 9 publications

Comparison of Image Transform-Based Features for Visual Speech Recognition in Clean and Corrupted Videos

Comparison of Image Transform-Based Features for Visual Speech Recognition in Clean and Corrupted Videos

Reliability-Based Large-Vocabulary Audio-Visual Speech Recognition

Robust Audio-Visual Speech Recognition Under Noisy Audio-Video Conditions

Contact Info

Product

Resources

About