Infrared and 3D Skeleton Feature Fusion for RGB-D Action Recognition

Boissiere, Alban Main de; Noumeir, Rita

doi:10.1109/access.2020.3023599

Cited by 43 publications

(23 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results are shown in Table 2. We compare our method with simple-based fusion, such as BI-LSTM [21] and FUSION [22], and attention-based fusion, such as MMT-M [16] and VPN(I3D) [14]. The evaluation results show that…”

Section: Comparisons With the State-of-the-artmentioning

confidence: 99%

Multimodal Feature Fusion Model For Rgb-D Action Recognition

Weiyao

Muqing

Zhao

et al. 2021

2021 IEEE International Conference on Multimedia &Amp; Expo Workshops (ICMEW)

View full text Add to dashboard Cite

Due to the variation and complexity of video data, human action recognition is still a challenging task. However, most of the methods based on RGB-D action recognition simply fuse the multimodal features, ignoring the potential semantic relationship between different modals. In this paper, we propose a multi-modal recognition model based on Bilinear Pooling and Attention Network (BPAN), which could effectively fuse multi-modal for RGB-D action recognition. Firstly, we adopt the efficient data preprocessing methods for RGB and skeleton. Then, a multimodal fusion network BPAN combining RGB and skeleton sequence is proposed, which could effectively compress the features of RGB and skeleton, and project them into potential subspace to obtain fusion features. In the end, a fully connected three-layer perceptron is adopted to obtain the final classification. Experimental results on two datasets demonstrate that our proposed method leads to a more favorable performance compared with the state-of-theart methods.

show abstract

Section: Comparisons With the State-of-the-artmentioning

confidence: 99%

Multimodal Feature Fusion Model For Rgb-D Action Recognition

Weiyao

Muqing

Zhao

et al. 2021

2021 IEEE International Conference on Multimedia &Amp; Expo Workshops (ICMEW)

View full text Add to dashboard Cite

show abstract

“…Other available video streams (RGB, infrared, depth) provide additional clues. Boissiere [ 31 ] proposed a modular network combining skeleton and infrared data. The pre-trained 2D CNN was used as a pose module to extract features from the skeleton data.…”

Section: Introductionmentioning

confidence: 99%

Body and Hand–Object ROI-Based Behavior Recognition Using Deep Learning

Byeon

Kim

Lee

et al. 2021

Sensors

View full text Add to dashboard Cite

Behavior recognition has applications in automatic crime monitoring, automatic sports video analysis, and context awareness of so-called silver robots. In this study, we employ deep learning to recognize behavior based on body and hand–object interaction regions of interest (ROIs). We propose an ROI-based four-stream ensemble convolutional neural network (CNN). Behavior recognition data are mainly composed of images and skeletons. The first stream uses a pre-trained 2D-CNN by converting the 3D skeleton sequence into pose evolution images (PEIs). The second stream inputs the RGB video into the 3D-CNN to extract temporal and spatial features. The most important information in behavior recognition is identification of the person performing the action. Therefore, if the neural network is trained by removing ambient noise and placing the ROI on the person, feature analysis can be performed by focusing on the behavior itself rather than learning the entire region. Therefore, the third stream inputs the RGB video limited to the body-ROI into the 3D-CNN. The fourth stream inputs the RGB video limited to ROIs of hand–object interactions into the 3D-CNN. Finally, because better performance is expected by combining the information of the models trained with attention to these ROIs, better recognition will be possible through late fusion of the four stream scores. The Electronics and Telecommunications Research Institute (ETRI)-Activity3D dataset was used for the experiments. This dataset contains color images, images of skeletons, and depth images of 55 daily behaviors of 50 elderly and 50 young individuals. The experimental results showed that the proposed model improved recognition by at least 4.27% and up to 20.97% compared to other behavior recognition methods.

show abstract

“…It differs from offline action detection where the sequence is studied in its entirety before temporal segments are proposed, then classified. Following the works of [8], we focus on infrared data from RGB-D cameras to further perpetuate their legitimacy as a stream candidate for action recognition and detection. Infrared from RGB-D cameras shows a strong potential for security and night-vision applications.…”

Section: Introductionmentioning

confidence: 99%

“…Infrared from RGB-D cameras shows a strong potential for security and night-vision applications. In [8], infrared yields great results for action recognition. In this work, we go one step further and use infrared as a ready-to-embed network for real-time online action detection.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bridging the gap between Human Action Recognition and Online Action Detection

Boissiere¹,

Noumeir²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Action recognition, early prediction, and online action detection are complementary disciplines that are often studied independently. Most online action detection networks use a pre-trained feature extractor, which might not be optimal for its new task. We address the task-specific feature extraction with a teacher-student framework between the aforementioned disciplines, and a novel training strategy. Our network, Online Knowledge Distillation Action Detection network (OKDAD), embeds online early prediction and online temporal segment proposal subnetworks in parallel. Low interclass and high intraclass similarity are encouraged during teacher training. Knowledge distillation to the OKDAD network is ensured via layer reuse and cosine similarity between teacher-student feature vectors. Layer reuse and similarity learning significantly improve our baseline which uses a generic feature extractor. We evaluate our framework on infrared videos from two popular datasets, NTU RGB+D (action recognition, early prediction) and PKU MMD (action detection). Unlike previous attempts on those datasets, our student networks perform without any knowledge of the future. Even with this added difficulty, we achieve state-ofthe-art results on both datasets. Moreover, our networks use infrared from RGB-D cameras, which we are the first to use for online action detection, to our knowledge.

show abstract

Infrared and 3D Skeleton Feature Fusion for RGB-D Action Recognition

Cited by 43 publications

References 52 publications

Multimodal Feature Fusion Model For Rgb-D Action Recognition

Multimodal Feature Fusion Model For Rgb-D Action Recognition

Body and Hand–Object ROI-Based Behavior Recognition Using Deep Learning

Bridging the gap between Human Action Recognition and Online Action Detection

Contact Info

Product

Resources

About