TORNADO: A Spatio-Temporal Convolutional Regression Network for Video Action Proposal

Zhu, Hongyuan; Vial, Romain; Lu, Shijian

doi:10.1109/iccv.2017.619

Cited by 58 publications

(37 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In their model, each complete activity instance is considered as a composition of three major stages, namely starting, course, and ending, and they introduced structured temporal pyramid pooling to produce a global representation of the entire proposal. Differently from previous methods, Zhu et al [179] proposed a framework that integrates the complementary spatial and temporal information into an end-to-end trainable system for video action proposal, and a novel and efficient path trimming method is proposed to handle untrimmed video by examining actionness and background score pattern without using extra detectors. To generalize R-CNN from 2D to 3D, Hou et al [52] proposed an end-to-end 3D CNN-based approach for action detection in videos.…”

Section: Cnn-based Approachmentioning

confidence: 99%

RGB-D-based human motion recognition with deep learning: A survey

Wang

Ogunbona

et al. 2018

Computer Vision and Image Understanding

354

183

View full text Add to dashboard Cite

Human motion recognition is one of the most important branches of human-centered research activities. In recent years, motion recognition based on RGB-D data has attracted much attention. Along with the development in artificial intelligence, deep learning techniques have gained remarkable success in computer vision. In particular, convolutional neural networks (CNN) have achieved great success for image-based tasks, and recurrent neural networks (RNN) are renowned for sequence-based problems. Specifically, deep learning methods based on the CNN and RNN architectures have been adopted for motion recognition using RGB-D data. In this paper, a detailed overview of recent advances in RGB-D-based motion recognition is presented. The reviewed methods are broadly categorized into four groups, depending on the modality adopted for recognition: RGB-based, depth-based, skeleton-based and RGB+D-based.As a survey focused on the application of deep learning to RGB-D-based motion recognition, we explicitly discuss the advantages and limitations of existing techniques. Particularly, we highlighted the methods of encoding spatialtemporal-structural information inherent in video sequence, and discuss potential directions for future research. body parts, in contrast with the few body parts that involved in gesture. An activity is composed by a sequence of actions. An interaction is a type of motion performed by two actors; one actor is human while the other may be human or an object. This implies that the interaction category will include human-human or human-object interaction."Hugging each other" and "playing guitar" are examples of these two kinds of interaction, respectively. Group activity is the most complex type of activity, and it may be a combination of gestures, actions and interactions. Necessarily, it involves more than two humans and from zero to multiple objects. Examples of group activities would include "two teams playing basketball" and "group meeting".Early research on human motion recognition was dominated by the analysis of still images or videos [2,144,132,99,44,176]. Most of these efforts used color and texture cues in 2D images for recognition. However, the task remains challenging due to problems posed by background clutter, partial occlusion, view-point, lighting changes, execution rate and biometric variation. This challenge remains even with current deep learning approaches [49,4].With the recent development of cost-effective RGB-D sensors, such as Microsoft Kinect TM and Asus Xtion TM , RGB-D-based motion recognition has attracted much attention. This is largely because the extra dimension (depth) is insensitive to illumination changes and includes rich 3D structural information of the scene. Additionally, 3D positions of body joints can be estimated from depth maps [114]. As a consequence, several methods based on RGB-D data have been proposed and the approach has proven to be a promising direction for human motion analysis.Several survey papers have summarized the research on human motion recognition...

show abstract

Section: Cnn-based Approachmentioning

confidence: 99%

RGB-D-based human motion recognition with deep learning: A survey

Wang

Ogunbona

et al. 2018

Computer Vision and Image Understanding

354

183

View full text Add to dashboard Cite

show abstract

“…These methods [9,39] are based on grouping techniques over low-level primitives such as color and motion, which reaffirms our intuition about the relevance of actors as a strong cue for the localization of the actions. The state of the art approaches for the generation of action proposals [44,41] are fully supervised based on a mix of convolutional and recurrent stages and supervised instance level tracking, respectively. Although these works generate action proposals of better quality, they do so at the cost of a significant amount of additional supervision.…”

Section: Quality Of Actor Proposalsmentioning

confidence: 99%

Guess where? Actor-supervision for spatiotemporal action localization

Escorcia

Dao

Jain³

et al. 2020

Computer Vision and Image Understanding

View full text Add to dashboard Cite

This paper addresses the problem of spatiotemporal localization of actions in videos. Compared to leading approaches, which all learn to localize based on carefully annotated boxes on training video frames, we adhere to a weakly-supervised solution that only requires a video class label. We introduce an actor-supervised architecture that exploits the inherent compositionality of actions in terms of actor transformations, to localize actions. We make two contributions. First, we propose actor proposals derived from a detector for human and non-human actors intended for images, which is linked over time by Siamese similarity matching to account for actor deformations. Second, we propose an actor-based attention mechanism that enables the localization of the actions from action class labels and actor proposals and is end-to-end trainable. Experiments on three human and non-human action datasets show actor supervision is state-of-the-art for weakly-supervised action localization and is even competitive to some fullysupervised alternatives.

show abstract

“…Singh et al [17] presented a deep-learning framework based on SSD with an efficient online algorithm to incrementally construct and label action tubes from the SSD frame level detections for real-time multiple spatio-temporal action localization and classification. Zhu et al [20] proposed a spatio-temporal convolutional network which consists of a temporal convolutional regression network and a spatial regression network by empowering convolutional LSTM with regression capability. Escorcia et al [12] developed an actor-supervised architecture that exploits the inherent compositionality of actions in terms of actor transformations to localize actions.…”

Section: Related Workmentioning

confidence: 99%

Discovering spatio-temporal action tubes

Yang

Tian

2019

Journal of Visual Communication and Image Representation

View full text Add to dashboard Cite

In this paper, we address the challenging problem of spatial and temporal action detection in videos. We first develop an effective approach to localize frame-level action regions through integrating static and kinematic information by the early-and late-fusion detection scheme. With the intention of exploring important temporal connections among the detected action regions, we propose a tracking-by-point-matching algorithm to stitch the discrete action regions into a continuous spatio-temporal action tube. Recurrent 3D convolutional neural network is used to predict action categories and determine temporal boundaries of the generated tubes. We then introduce an action footprint map to refine the candidate tubes based on the action-specific spatial characteristics preserved in the convolutional layers of R3DCNN. In the extensive experiments, our method achieves superior detection results on the three public benchmark datasets: UCFSports, J-HMDB and UCF101.

show abstract

TORNADO: A Spatio-Temporal Convolutional Regression Network for Video Action Proposal

Cited by 58 publications

References 21 publications

RGB-D-based human motion recognition with deep learning: A survey

RGB-D-based human motion recognition with deep learning: A survey

Guess where? Actor-supervision for spatiotemporal action localization

Discovering spatio-temporal action tubes

Contact Info

Product

Resources

About