Combining Expert Strategies in Multimodal Classification

Abstract

In multimodal classification, we look for a set of strategies for mining and exploiting the most informative modalities for a given situation. These strategies are computations performed by the algorithms. In this paper, we propose to consider strategies as advice given to an algorithm by “expert.” There can be several classification strategies. Each strategy makes different assumptions regarding the fidelity of a sensor modality and uses different data to arrive at its estimates. Each strategy may place different trust in a sensor at different times, and each may be better in different situations. In this paper, we introduce a novel algorithm for combining expert strategies to achieve robust classification performance in a multimodal setting. We provide experimental results using real world examples to demonstrate the efficacy of the proposed algorithm.