Improving Multi-Modal Learning with Uni-Modal Teachers

Abstract

Learning multi-modal representations is an essential step towards real-world robotic applications, and various multi-modal fusion models have been developed for this purpose. However, we observe that existing models, whose objectives are mostly based on joint training, often suffer from learning inferior representations of each modality. We name this problem Modality Failure, and hypothesize that the imbalance of modalities and the implicit bias of common objectives in fusion method prevent encoders of each modality from sufficient feature learning. To this end, we propose a new multi-modal learning method, Uni-Modal Teacher, which combines the fusion objective and uni-modal distillation to tackle the modality failure problem. We show that our method not only drastically improves the representation of each modality, but also improves the overall multi-modal task performance. Our method can be effectively generalized to most multi-modal fusion approaches. We achieve more than 3% improvement on the VGGSound audio-visual classification task, as well as improving performance on the NYU depth V2 RGB-D image segmentation task.

A multimodal resource balancing approach based on cross-attention and bidrectional associative memory networks

“…Wang et al [14] used a gradient mixing method to solve the problem of varying generalization speeds for different modal data. Du et al [15] used single mode self distillation and distillation fusion features to solve the problem of modal imbalance and implicit bias of common targets in fusion methods. Peng [16] proposed using dynamic gradient modulation for adaptive control, optimizing the contribution of each mode to the learning objectives by monitoring their differences, in order to achieve modal equilibrium.…”

PMR: Prototypical Modal Rebalance for Multimodal Learning

“…Several recent studies [5,40,41] have shown that many multimodal DNNs cannot achieve better performance compared to the best single-modal DNNs. Wang et al [40] found that different modalities overfit and generalize at different rates and thus obtain suboptimal solutions when jointly training them using a unified optimization strategy.…”

“…Several approaches have been proposed recently to deal with the modal imbalance problem [5,29,40,44]. Wang et al [40] used additional classifiers for each modality and its fusion modality and then optimized the gradient mixing problem they introduced to obtain better weights for each branch.…”

“…Wang et al [40] used additional classifiers for each modality and its fusion modality and then optimized the gradient mixing problem they introduced to obtain better weights for each branch. Du et al [5] tried to improve uni-modal performance by distilling knowledge from well-trained models. However, these approaches introduce more model structure and computational effort, which makes the training process more complex and expensive.…”

“…However, we find out through experiments that the dominant modality not only suppresses the learning rates of other modalities [29] but also interferes with their update direction, which makes it hard to improve the performance of slow-learning modalities. Moreover, existing methods either inevitably bring additional model structures [5,40] or are limited by specific fusion methods [29,42], which limit their application scenarios.…”