Audio-visual speech enhancement with AVCDCN (audio-visual codebook dependent cepstral normalization)

Deligne, Sabine; Potamianos, Gerasimos; Neti, C.

doi:10.1109/sam.2002.1191001

Cited by 22 publications

(25 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Progress in addressing some or all of these questions can also benefit other areas where joint audio and visual speech processing is suitable [139], such as speaker identification and verification [49], [66], [109], [136], [140][141][142], visual text-to-speech [143][144][145] speech event detection [146] video indexing and retrieval [147], speech enhancement [102], [104], coding [148], signal separation [149], [150], and speaker localization [151][152][153]. Improvements in these areas will result in more robust and natural humancomputer interaction.…”

Section: Summary and Discussionmentioning

confidence: 99%

“…Audio feature enhancement on basis of either visual input [14], [101], or concatenated audio-visual features [102][103][104] (4), while seeking the best discrimination among the speech classes of interest. In [99], LDA is followed by an MLLT rotation of the feature vector to improve statistical data modeling by means of Gaussian mixture emission probability densities with diagonal covariances, as in (1).…”

Section: A Feature Fusionmentioning

confidence: 99%

See 1 more Smart Citation

Recent advances in the automatic recognition of audiovisual speech

et al. 2003

Self Cite

View full text Add to dashboard Cite

Abstract-Visual speech information from the speaker's mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability into the human computer interface. In this paper, we review the main components of audio-visual automatic speech recognition and present novel contributions in two main areas: First, the visual front end design, based on a cascade of linear image transforms of an appropriate video region-of-interest, and subsequently, audio-visual speech integration. On the later topic, we discuss new work on feature and decision fusion combination, the modeling of audio-visual speech asynchrony, and incorporating modality reliability estimates to the bimodal recognition process. We also briefly touch upon the issue of audiovisual speaker adaptation. We apply our algorithms to three multi-subject bimodal databases, ranging from small-to largevocabulary recognition tasks, recorded at both visually controlled and challenging environments. Our experiments demonstrate that the visual modality improves automatic speech recognition over all conditions and data considered, however less so for visually challenging environments and large vocabulary tasks.

show abstract

Section: Summary and Discussionmentioning

confidence: 99%

Section: A Feature Fusionmentioning

confidence: 99%

Recent advances in the automatic recognition of audiovisual speech

et al. 2003

Self Cite

View full text Add to dashboard Cite

show abstract

“…MSE criterion between true and estimated outputs V(k) andV(k) is used as the performance measure in training. This is the same criterion as what is used in contrast function (8). Networks were trained using the LevenbergMarquardt algorithm [38] and via early stopping based on validation subset to avoid over-fit.…”

Section: Building Audio-visual Modelsmentioning

confidence: 99%

“…The complementary information of AV data is truly adopted in audio-visual speech recognition (AVSR) in either of early-(feature), middle-(model), or late-(decoding) stage fusion schemes to enhance robustness against acoustic distortions. In recent years (since 2001 [7]), researchers have proposed methods based on exploiting the coherent component of AV processes for applicable tasks like speech enhancement [7][8][9], acoustic feature enhancement [10], visual voice activity detection (VVAD) [11], and AV source separation (AVSS) [11][12][13][14][15][16][17][18][19][20][21][22][23][24].…”

Section: Introductionmentioning

confidence: 99%

Audio visual speech source separation via improved context dependent association model

Kazemi

Boostani

Sobhanmanesh

2014

EURASIP J. Adv. Signal Process.

View full text Add to dashboard Cite

In this paper, we exploit the non-linear relation between a speech source and its associated lip video as a source of extra information to propose an improved audio-visual speech source separation (AVSS) algorithm. The audio-visual association is modeled using a neural associator which estimates the visual lip parameters from a temporal context of acoustic observation frames. We define an objective function based on mean square error (MSE) measure between estimated and target visual parameters. This function is minimized for estimation of the de-mixing vector/filters to separate the relevant source from linear instantaneous or time-domain convolutive mixtures. We have also proposed a hybrid criterion which uses AV coherency together with kurtosis as a non-Gaussianity measure. Experimental results are presented and compared in terms of visually relevant speech detection accuracy and output signal-to-interference ratio (SIR) of source separation. The suggested audio-visual model significantly improves relevant speech classification accuracy compared to existing GMM-based model and the proposed AVSS algorithm improves the speech separation quality compared to reference ICA-and AVSS-based methods.

show abstract

“…In recent years, the visual modality has also been exploited for speech enhancement in (background) noise (Girin et al, 2001;Deligne et al, 2002;Potamianos et al, 2003b), and more generally for speech source separation, i.e., for the extraction of a speech signal from complex mixtures using several microphones, for both linear instantaneous mixtures (Sodoyer et al, 2002(Sodoyer et al, , 2004) and convolutive mixtures (Wang et al, 2005;Rivet et al, 2007).…”

Section: A Context: Audio-visual Speech Processingmentioning

confidence: 99%

A study of lip movements during spontaneous dialog and its application to voice activity detection

Sodoyer

Rivet

Girin

et al. 2009

The Journal of the Acoustical Society of America

View full text Add to dashboard Cite

Running title: Voice activity detection based on lip movementsThis paper presents a quantitative and comprehensive study of the lip movements of a given speaker in different speech / non speech contexts, with a particular focus on silences (i.e., when no sound is produced by the speaker). The aim is to characterize the relationship between "lip activity" and "speech activity", and then to use visual speech information as a Voice Activity Detector (VAD). To this aim, an original audio-visual corpus was recorded with two speakers involved in a face-to-face spontaneous dialog, although being in separate rooms. Each speaker communicated with the other using a microphone, a camera, a screen, and headphones. This system was used to capture separate audio stimuli for each speaker and to monitor each

show abstract

Audio-visual speech enhancement with AVCDCN (audio-visual codebook dependent cepstral normalization)

Cited by 22 publications

References 5 publications

Recent advances in the automatic recognition of audiovisual speech

Recent advances in the automatic recognition of audiovisual speech

Audio visual speech source separation via improved context dependent association model

A study of lip movements during spontaneous dialog and its application to voice activity detection

Contact Info

Product

Resources

About