Heterogeneous Non-Local Fusion for Multimodal Activity Recognition

Abstract

In this work, we investigate activity recognition using multimodal inputs from heterogeneous sensors. Activity recognition is commonly tackled from a single-modal perspective using videos. In case multiple signals are used, they come from the same homogeneous modality, e.g. in the case of color and optical flow. Here, we propose an activity network that fuses multimodal inputs coming from completely different and heterogeneous sensors. We frame such a heterogeneous fusion as a non-local operation. The observation is that in a non-local operation, only the channel dimensions need to match. In the network, heterogeneous inputs are fused, while maintaining the shapes and dimensionalities that fit each input. We outline both asymmetric fusion, where one modality serves to enforce the other, and symmetric fusion variants. To further promote research into multimodal activity recognition, we introduce GloVid, a first-person activity dataset captured with video recordings and smart glove sensor readings. Experiments on GloVid show the potential of heterogeneous non-local fusion for activity recognition, outperforming individual modalities and standard fusion techniques. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding.

Simulation-Driven Design of Smart Gloves for Gesture Recognition

“…Combining video inputs with smart glove data in a non-local manner allows us to recognize activities, even when they are not visible on camera. [11] of interest in the beginning of a video-sensor clip and could recognize the motion-specific sensor reading in the end of the video-sensor clip, achieving temporal non-locality. As seen in Figure 3.2, two modalities, smart gloves and videos, can provide information hidden from the other modality.…”

Social Fabric: Tubelet Compositions for Video Relation Detection

“…Stage 1 Stage 2 P@1 P@5 P@10 mAP R@50 R@100 60.72 46. 40 [48] for textual sequence-to-sequence tasks and recently adopted in video tasks [4,14,15]. Here, we investigate their potential for interaction detection.…”

FedNKD: A Dependable Federated Learning Using Fine-tuned Random Noise and Knowledge Distillation

“…At present, multimedia retrieval needs to obtain the content needed by users from very rich video, pictures and text data [2,[4][5][6]24]. Image classification model is an important part of multimedia retrieval task.…”