Stream weight tuning in dynamic Bayesian networks

Abstract

In this paper we present a family of algorithms for estimating stream weights for dynamic Bayesian networks with multiple observation streams. For the 2 stream case, we present a weight tuning algorithm optimal in the minimum classification error sense. We compare the algorithms to brute-force search where feasible, as well as to previously published algorithms and show that the algorithms perform as well as brute-force search and outperform previously published algorithms. We test the stream weight tuning algorithm in the context of speech recognition with distinctive feature tandem models. We analyze how the criterion used for weight tuning differs from the standard word error rate criterion used in speech recognition.

Learning Dynamic Stream Weights For Coupled-HMM-based Audio-visual Speech Recognition

“…The auxiliary function in (15) can now be optimized by separately optimizing for each frame . By applying (1) to (15) we get the following expression for the auxiliary function at each : (19) independent of (20) where and are the single-modality acoustical and visual states composing the coupled state .…”

“…While in some prior works, the stream weight for the whole dataset has been set to a fixed value, which was found using grid search, e.g., [11], [14] or using other tuning algorithms, e.g., [15], some authors have assumed that the stream weight is a model parameter and have estimated it using generative [16] or discriminative [17], [18] criteria. In real scenarios, however, the reliability of the audio and video modality can vary quickly, even on the frame level, and such fixed or model-dependent estimation might lead to worse results than using Bayes fusion, i.e., equal weights.…”