Multi-View Action Recognition using Contrastive Learning

Shah, Ketul; Shah, Anshul; Lau, Chun Pong; Melo, Celso M. de; Chellapp, Rama

doi:10.1109/wacv56688.2023.00338

Cited by 23 publications

(11 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can be seen from the Table IV that the ArViAU method obtains the best accuracy on the benchmark datasets of MICAGes (94.03%), IXMAS (86.75%), MuHAVi (95.35%), and NUMA (93.19%). In addition, these results outperform the SOTA methods of D A + ELM + aug [42], Shah et al [45]. For IXMAS dataset, our proposed method with the case of ArViAVR ((ArVi-MoCoGAN+C3D); AVR) has the accuracy of 82.03%.…”

Section: The Experimental Results Inmentioning

confidence: 59%

An End-to-End Model of ArVi-MoCoGAN and C3D with Attention Unit for Arbitrary-view Dynamic Gesture Recognition

Doan,

Luong,

Pham

2024

IJACSA

View full text Add to dashboard Cite

Human gesture recognition is an attractive research area in computer vision with many applications such as humanmachine interaction, virtual reality, etc. Recent deep learning techniques have been efficiently applied for gesture recognition, but they require a large and diverse amount of training data. In fact, the available gesture datasets contain mostly static gestures and/or certain fixed viewpoints. Some contain dynamic gestures, but they are not diverse in poses and viewpoints. In this paper, we propose a novel end-to-end framework for dynamic gesture recognition from unknown viewpoints. It has two main components: (1) an efficient GAN-based architecture, named ArVi-MoCoGAN; (2) the gesture recognition component, which contains C3D backbones and an attention unit. ArVi-MoCoGAN aims at generating videos at multiple fixed viewpoints from a real dynamic gesture at an arbitrary viewpoint. It also returns the probability that a real arbitrary view gesture belongs to which of the fixed-viewpoint gestures. These outputs of ArVi-MoCoGAN will be processed in the next component to improve the arbitrary view recognition performance through multi-view synthetic gestures. The proposed system is extensively analyzed and evaluated on four standard dynamic gesture datasets. The experimental results of our proposed method are better than the current solutions, from 1% to 13.58% for arbitrary view gesture recognition and from 1.2% to 7.8% for single view gesture recognition.

show abstract

Section: The Experimental Results Inmentioning

confidence: 59%

An End-to-End Model of ArVi-MoCoGAN and C3D with Attention Unit for Arbitrary-view Dynamic Gesture Recognition

Doan,

Luong,

Pham

2024

IJACSA

View full text Add to dashboard Cite

show abstract

“…Moreover, we note that the performance for the models trained on the ground data tend to perform better than the models trained in the air data, thus emphasizing the different characteristics between these perspectives. We recently began exploring a method that relies on intermediate 3D representations to support robust generalization from the ground to the air perspective [9].…”

Section: Closing the Domain Gapmentioning

confidence: 99%

Synthetic data for automatic target recognition from small drones

Melo

Conover

Poster

et al. 2023

Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications

View full text Add to dashboard Cite

Automatic target recognition (ATR) technology is likely to play an increasingly prevalent role in maintaining situational awareness in the modern battlefield. Progress in deep learning has enabled considerable progress in the development of ATR algorithms; however, these algorithms require large amounts of high-quality annotated data to train and that is often the main bottleneck. Synthetic data offers a potential solution to this problem, especially given recent proliferation of tools and techniques to synthesize custom data. Here, we focus on ATR, in the visible domain, from the perspective of a small drone, which represents a domain of growing importance to the Army. We describe custom simulators built to support synthetic data for multiple targets in a variety of environments. We describe a field experiment where we compared a baseline (YOLOv5) model, trained on off-the-shelf large generic public datasets, with a model augmented with specialized synthetic data. We deployed the models on a VOXL platform in a small drone. Our results showed a considerable boost in performance when using synthetic data of over 40% in target detection accuracy (average precision with at least 50% overlap). We discuss the value of synthetic data for this domain, the opportunities it creates, but also the novel challenges it introduces.

show abstract

“…This concept has various applications in downstream tasks, including pose estimation [59], robotics [20,53], and 3D detection [34,41]. Some early approaches [21,50,56] utilize synchronized multi-view data to address the challenge of viewagnostic representation. However, the high cost of such data makes these approaches barely scalable for broader applications.…”

Section: Related Workmentioning

confidence: 99%

POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-view World

Xu,

Zheng,

Jin

2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

We humans are good at translating third-person observations of hand-object interactions (HOI) into an egocentric view. However, current methods struggle to replicate this ability of view adaptation from third-person to first-person. Although some approaches attempt to learn view-agnostic representation from large-scale video datasets, they ignore the relationships among multiple third-person views. To this end, we propose a Prompt-Oriented View-agnostic learning (POV) framework in this paper, which enables this view adaptation with few egocentric videos. Specifically, We introduce interactive masking prompts at the frame level to capture finegrained action information, and view-aware prompts at the token level to learn view-agnostic representation. To verify our method, we establish two benchmarks for transferring from multiple thirdperson views to the egocentric view. Our extensive experiments on these benchmarks demonstrate the efficiency and effectiveness of our POV framework and prompt tuning techniques in terms of view adaptation and view generalization. Our code is available at https://github.com/xuboshen/pov_acmmm2023. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding; Scene understanding; Transfer learning.

show abstract

Multi-View Action Recognition using Contrastive Learning

Cited by 23 publications

References 57 publications

An End-to-End Model of ArVi-MoCoGAN and C3D with Attention Unit for Arbitrary-view Dynamic Gesture Recognition

An End-to-End Model of ArVi-MoCoGAN and C3D with Attention Unit for Arbitrary-view Dynamic Gesture Recognition

Synthetic data for automatic target recognition from small drones

POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-view World

Contact Info

Product

Resources

About