Graph-based High-order Relation Modeling for Long-term Action Recognition

Zhou, Jiaming; Lin, Kun-Yu; Li, Haoxin; Zheng, Wei‐Shi

doi:10.1109/cvpr46437.2021.00887

Cited by 32 publications

(9 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(4) Construct the high-order semantic relationship between joint points [188,189] . For the higher-order association between joints, such as the association between multi-view joints, but the current methods appropriate modeling methods.…”

Section: Discussionmentioning

confidence: 99%

A Survey of Human Action Recognition and Posture Prediction

Cheung

et al. 2022

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

Human action recognition and posture prediction aim to recognize and predict respectively the action and postures of persons in videos. They are both active research topics in computer vision community, which have attracted considerable attention from academia and industry. They are also the precondition for intelligent interaction and human-computer cooperation, and they help the machine perceive the external environment. In the past decade, tremendous progress has been made in the field, especially after the emergence of deep learning technologies.Hence, it is necessary to make a comprehensive review of recent developments. In this paper, firstly, we attempt to present the background, and then discuss research progresses. Secondly, we introduce datasets, various typical feature representation methods, and explore advanced human action recognition and posture prediction algorithms.Finally, facing the challenges in the field, this paper puts forward the research focus, and introduces the importance of action recognition and posture prediction by taking interactive cognition in self-driving vehicle as an example.

show abstract

Section: Discussionmentioning

confidence: 99%

A Survey of Human Action Recognition and Posture Prediction

Cheung

et al. 2022

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…Apart from the above-mentioned architectures, i.e., twostream 2D CNN, RNN, 3D CNN, and Transformer, there are also some other frameworks designed for HAR using RGB videos, such as Convolutional Gated Restricted Boltzmann Machines [178], Graph-based Modeling [179], [180], and 4D CNNs [181].…”

Section: Transformer-based Methodsmentioning

confidence: 99%

Human Action Recognition From Various Data Modalities: A Review

Sun

Rahmani

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

283

View full text Add to dashboard Cite

Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.

show abstract

“…Pretraning Dataset Pretraining Samples Accuracy(↑) VideoGraph [24] Kinetics 306K 69.50 Timeception [23] Kinetics 306K 71.30 GHRM [57] Kinetics 306K 75.50 Distant Supervision [32] HowTo100M 136M 89.90 ViS4mer Kinetics 495K 88.17…”

Section: Modelmentioning

confidence: 99%

Long Movie Clip Classification with State-Space Video Models

Islam¹,

Bertasius²

2022

Preprint

View full text Add to dashboard Cite

Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Because of this, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning capabilities. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the recently introduced structured state-space sequence (S4) layer. Our model uses a standard Transformer encoder for short-range spatiotemporal feature extraction, and a multi-scale temporal S4 decoder for subsequent long-range temporal reasoning. By progressively reducing the spatiotemporal feature resolution and channel dimension at each decoder layer, ViS4mer learns complex long-range spatiotemporal dependencies in a video. Furthermore, ViS4mer is 2.63× faster and requires 8× less GPU memory than the corresponding pure self-attention-based model. Additionally, ViS4mer achieves state-of-the-art results in 7 out of 9 long-form movie video classification tasks on the LVU benchmark. Furthermore, we also show that our approach successfully generalizes to other domains, achieving competitive results on the Breakfast and the COIN procedural activity datasets. The code will be made publicly available. 1 Recently, several Transformer models [6,50] have been shown to perform well on long-range video understanding tasks. However, due to the quadratic cost of standard self-attention, these models are 1 https://github.com/md-mohaiminul/ViS4merPreprint. Under review.

show abstract

Graph-based High-order Relation Modeling for Long-term Action Recognition

Cited by 32 publications

References 21 publications

A Survey of Human Action Recognition and Posture Prediction

A Survey of Human Action Recognition and Posture Prediction

Human Action Recognition From Various Data Modalities: A Review

Long Movie Clip Classification with State-Space Video Models

Contact Info

Product

Resources

About