MuMu: Cooperative Multitask Learning-Based Guided Multimodal Fusion

Abstract

Multimodal sensors (visual, non-visual, and wearable) can provide complementary information to develop robust perception systems for recognizing activities accurately. However, it is challenging to extract robust multimodal representations due to the heterogeneous characteristics of data from multimodal sensors and disparate human activities, especially in the presence of noisy and misaligned sensor data. In this work, we propose a cooperative multitask learning-based guided multimodal fusion approach, MuMu, to extract robust multimodal representations for human activity recognition (HAR). MuMu employs an auxiliary task learning approach to extract features specific to each set of activities with shared characteristics (activity-group). MuMu then utilizes activity-group-specific features to direct our proposed Guided Multimodal Fusion Approach (GM-Fusion) for extracting complementary multimodal representations, designed as the target task. We evaluated MuMu by comparing its performance to state-of-the-art multimodal HAR approaches on three activity datasets. Our extensive experimental results suggest that MuMu outperforms all the evaluated approaches across all three datasets. Additionally, the ablation study suggests that MuMu significantly outperforms the baseline models (p

Action recognition based on multimode fusion for VR online platform

“…Using our VCMA dataset, the highest average accuracy of 99.81% is achieved by the augmented key point data and sensor data. The highest accuracy of the UTD-MHAD dataset is 97.6% (Islam 2022), and the highest accuracy of the Berkeley MHAD dataset is 99.6% (Ahmad and Khan 2020) using depth and inertial sensor data. Although the modalities used are different, they are all multimodal action recognition with sensor data.…”

Multi-Modal Representation via Contrastive Learning with Attention Bottleneck Fusion and Attentive Statistics Features

“…Moreover, hybrid fusion techniques have also been explored, combining feature-level and decision-level fusion approaches [17][18][19]. These techniques aim to leverage the benefits of both strategies by fusing low-level sensory features and high-level decision outputs.…”

Feature representation learning for human activity recognition

“…Although these modalities also bring their own limitations and challenges, especially for some real-world applications, they have a huge potential to contribute to activity recognition performance as well as to unlock the full capabilities of the skeleton and inertial modalities. Besides, future work could benefit from using more advanced and sophisticated encoder architectures that may enable additional cross-modal fusion strategies [94,95]. While different supervised and SSL methods have been adapted to HAR in research studies and this dissertation, in particular, there is a limited amount of studies that are focused on analyzing representations produced by these algorithms.…”