Multi-modal Transformer for Indoor Human Action Recognition

Do, Jeonghyeok; Kim, Munchurl

doi:10.23919/iccas55662.2022.10003914

Cited by 2 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As another early fusion method, early concatenation entails concatenating token embedding sequences from multiple modalities and inputting them into transformer layers. Studies in [36], [37], and [42] employed this fusion method with RGB and skeleton sequences for HAR. Early concatenation offers the benefit of relative simplicity compared to other methods.…”

Section: B Multimodal Fusion For Har 1) Transformer-based Multimodali...mentioning

confidence: 99%

“…1) Recognition of fine-grained HOI : Most existing HAR studies [12], [13], [36], [37], [42] that fuse skeleton and RGB modalities have mainly concentrated on recognizing broad interaction categories. These studies have assessed datasets like NTU RGB+D [27], which contain only coarse-grained HOI that can be accurately classified using high-quality skeleton modality features alone.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SSRT: A Sequential Skeleton RGB Transformer to Recognize Fine-Grained Human-Object Interactions and Action Recognition

2023

View full text Add to dashboard Cite

Combining skeleton and RGB modalities in human action recognition (HAR) has garnered attention due to their ability to complement each other. However, previous studies did not address the challenge of recognizing fine-grained human-object interaction (HOI). To tackle this problem, this study introduces a new transformer-based architecture called Sequential Skeleton RGB Transformer (SSRT), which fuses skeleton and RGB modalities. First, SSRT leverages the strength of Long Short-Term Memory (LSTM) and a multi-head attention mechanism to extract high-level features from both modalities. Subsequently, SSRT employs a two-stage fusion method, including transformer cross-attention fusion and softmax layer late score fusion, to effectively integrate the multimodal features. Aside from evaluating the proposed method on fine-grained HOI, this study also assesses its performance on two other action recognition tasks: general HAR and cross-dataset HAR. Furthermore, this study conducts a performance comparison between a HAR model using single-modality features (RGB and skeleton) alongside multimodality features on all three action recognition tasks. To ensure a fair comparison, comparable state-of-the-art transformer architectures are employed for both the single-modality HAR model and SSRT. In terms of modality, SSRT outperforms the best-performing single-modality HAR model on all three tasks, with accuracy improved by 9.92% on fine-grained HOI recognition, 6.73% on general HAR, and 11.08% on cross-dataset HAR. Additionally, the proposed fusion model surpasses state-of-the-art multimodal fusion techniques like Transformer Early Concatenation, with an accuracy improved by 6.32% on fine-grained HOI recognition, 4.04% on general HAR, and 6.56% on cross-dataset.INDEX TERMS Multimodality fusion, human action recognition, fine-grained actions, transformer crossattention fusion.

show abstract

Section: B Multimodal Fusion For Har 1) Transformer-based Multimodali...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SSRT: A Sequential Skeleton RGB Transformer to Recognize Fine-Grained Human-Object Interactions and Action Recognition

2023

View full text Add to dashboard Cite

show abstract

“…The video backbone consists of 3D CNNs to extract spatio-temporal features, and the pose backbone contains a spatio-temporal GCN. Do et al (2022) propose a Multimodal Transformer (MMT) to use RGB and skeleton data of only eight input frames. Using the transformer-based structure, MMT can capture the correlation between non-local joints in skeleton data modality.…”

Section: Single-stream Various Papers Introduce Skeleton As Attention...mentioning

confidence: 99%

Multimodal vision-based human action recognition using deep learning: a review

Shafizadegan,

Naghsh-Nilchi,

Shabaninia

2024

Artif Intell Rev

View full text Add to dashboard Cite

Vision-based Human Action Recognition (HAR) is a hot topic in computer vision. Recently, deep-based HAR has shown promising results. HAR using a single data modality is a common approach; however, the fusion of different data sources essentially conveys complementary information and improves the results. This paper comprehensively reviews deep-based HAR methods using multiple visual data modalities. The main contribution of this paper is categorizing existing methods into four levels, which provides an in-depth and comparable analysis of approaches in various aspects. So, at the first level, proposed methods are categorized based on the employed modalities. At the second level, methods categorized in the first level are classified based on the employment of complete modalities or working with missing modalities at the test time. At the third level, complete and missing modality branches are categorized based on existing approaches. Finally, similar frameworks in the third category are grouped together. In addition, a comprehensive comparison is provided for publicly available benchmark datasets, which helps to compare and choose suitable datasets for a task or to develop new datasets. This paper also compares the performance of state-of-the-art methods on benchmark datasets. The review concludes by highlighting several future directions.

show abstract

Multi-modal Transformer for Indoor Human Action Recognition

Cited by 2 publications

References 26 publications

SSRT: A Sequential Skeleton RGB Transformer to Recognize Fine-Grained Human-Object Interactions and Action Recognition

SSRT: A Sequential Skeleton RGB Transformer to Recognize Fine-Grained Human-Object Interactions and Action Recognition

Multimodal vision-based human action recognition using deep learning: a review

Contact Info

Product

Resources

About