Modality Compensation Network: Cross-Modal Adaptation for Action Recognition

Abstract

With the prevalence of RGB-D cameras, multimodal video data have become more available for human action recognition. One main challenge for this task lies in how to effectively leverage their complementary information. In this work, we propose a Modality Compensation Network (MCN) to explore the relationships of different modalities, and boost the representations for human action recognition. We regard RGB/optical flow videos as source modalities, skeletons as auxiliary modality. Our goal is to extract more discriminative features from source modalities, with the help of auxiliary modality. Built on deep Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) networks, our model bridges data from source and auxiliary modalities by a modality adaptation block to achieve adaptive representation learning, that the network learns to compensate for the loss of skeletons at test time and even at training time. We explore multiple adaptation schemes to narrow the distance between source and auxiliary modal distributions from different levels, according to the alignment of source and auxiliary data in training. In addition, skeletons are only required in the training phase. Our model is able to improve the recognition performance with source data when testing. Experimental results reveal that MCN outperforms stateof-the-art approaches on four widely-used action recognition benchmarks.

Human Action Recognition from Various Data Modalities: A Review

“…Specifically, the teacher network was trained with RGB videos, providing supervision information for the student network handling skeleton data. Song et al [397] proposed a Modality Compensation Network (MCN) taking advantage of the skeleton modality to compensate the feature learning of the RGB modality with adaptive representation learning, and a modality adaptation block with residual feature learning was designed to bridge information between modalities.…”

Batch Entropy Supervised Convolutional Neural Networks for Feature Extraction and Harmonizing for Action Recognition

“…Due to the heterogeneous, cluttered and dynamic background network gets confused with the action classes. Several attention and skeleton modality based approaches have been employed to generate discriminative features [15] [14]. Recently some works focused on extracting highly discriminative features using Extreme Learning Machine (ELM) [66].…”

“…Because of the noisy motion and appearance of unimportant infor-mation, close-fitted inter-class discrimination and extensive intra-class discrimination make this task quite challenging [8] [9]. Decreasing the feature discrimination between the intra-class and increasing the discrimination between interclass features can be one of the effective solutions for the concerned issue [14].…”

“…Numerous, action recognition techniques based on depth information along with RGB frames using RGB-D dataset for 3D action recognition [65] [71], attention mechanism [33] - [40] and skeleton modality [14] [70] [72] have been introduced to deal with the addressed problem. These published methods demand to pre-process the data, which increases the latency and time complexity during prediction.…”

“…These published methods demand to pre-process the data, which increases the latency and time complexity during prediction. Inspired by the work in [14], rather than pre-processing the data, we focus on increasing and then harmonizing the feature dis-FIGURE 1 Overall procedure: A visual representation of the end-to-end process. Left part shows an example of a batch input with conceptual histogram representation.…”