Multi-level Fusion of Audio and Visual Features for Speaker Identification

Abstract

This paper explores the fusion of audio and visual evidences through a multi-level hybrid fusion architecture based on dynamic Bayesian network (DBN), which combines model level and decision level fusion to achieve higher performance. In model level fusion, a new audiovisual correlative model (AVCM) based on DBN is proposed, which describes both the intercorrelations and loose timing synchronicity between the audio and video streams. The experiments on the CMU database and our own homegrown database both demonstrate that the methods can improve the accuracies of audiovisual bimodal speaker identification at all levels of acoustic signal-to-noiseratios (SNR) from 0dB to 30dB with varying acoustic conditions.

Analysis of multimodal fusion techniques for audio-visual speech recognition

“…Кроме того, существует и другой способ, который находится между ранней и поздней интеграцией и называется промежуточной интегра-цией (в некоторых источниках его относят к ранней интеграции). Также можно комплексировать два спо-соба интеграции, выполняя объединение одновременно на двух уровнях, что называют гибридным под-ходом [52]. Далее эти способы описаны более подробно с анализом их преимуществ и недостатков.…”

Audio Visual Person Authentication by Multiple Nearest Neighbor Classifiers

“…However, proper fusion of multiple biometrics remains a challenging and crucial task. Amongst the various multimodal biometric methods reported so far, Audio-Visual person authentication [3][4][5][6][7][8][17] offers some more unique advantages. First of all, it uses two biometrics (speech and face image), which people share quite comfortably in everyday life.…”

“…A few recent methods (e.g. [8]) proposed a feature level fusion as well. For the speech mode, various text-independent speaker recognition methods, using GMM [11] or VQ [10], are predominantly used.…”

Weight Estimation for Audio-Visual Multi-level Fusion in Bimodal Speaker Identification

“…It is generally agreed that the model level fusion gives better performance because it can capture the potentially useful coupling or conditional dependence between audio visual modalities [2][3][4][5]. However, in a very noisy environment, the performance of model level fusion may not be as good as that of decision level fusion [2,5].…”

“…It combines model level and decision level fusion to achieve improved performance [5]. In such a strategy, the fusion weights are of great importance as they must capture the reliability of inputs which may vary dynamically.…”