DeepCAMP: Deep Convolutional Action & Attribute Mid-Level Patterns

Abstract

The recognition of human actions and the determination of human attributes are two tasks that call for fine-grained classification. Indeed, often rather small and inconspicuous objects and features have to be detected to tell their classes apart. In order to deal with this challenge, we propose a novel convolutional neural network that mines mid-level image patches that are sufficiently dedicated to resolve the corresponding subtleties. In particular, we train a newly designed CNN (DeepPattern) that learns discriminative patch groups. There are two innovative aspects to this. On the one hand we pay attention to contextual information in an original fashion. On the other hand, we let an iteration of feature learning and patch clustering purify the set of dedicated patches that we use. We validate our method for action classification on two challenging datasets: PASCAL VOC 2012 Action and Stanford 40 Actions, and for attribute recognition we use the Berkeley Attributes of People dataset. Our discriminative mid-level mining CNN obtains state-of-theart results on these datasets, without a need for annotations about parts and poses.

Encouraging LSTMs to Anticipate Actions Very Early

“…1). This is in contrast to existing methods that typically extract either global representations for the entire image [6,45,46,7] or video sequence [38,16], thus not focusing on the action itself, or localize the feature extraction process to the action itself via dense trajectories [43,42,9], optical flow [8,45,20] or actionness [44,3,14,48,39,24], thus failing to exploit contextual information. To the best of our knowledge, only two-stream networks [30,8,4,22] have attempted to jointly leverage both information types by making use of RGB frames in conjunction with optical flow to localize the action.…”

“…Most recent action approaches extract global representations for the entire image [6,46,7] or video sequence [38,16]. As such, these methods do not truly focus on the actions of interest, but rather compute a context-aware representation.…”

A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences

“…[119] focuses on the changes that an action brings into the environment and propose a siamese CNN architecture to fuse precondition and effect information from the environment. [20] proposes a CNN which uses mid-level discriminative visual elements. The method, called DeepPattern, is able to learn discriminative patches by exploring human body parts as well as scene context.…”

Deep Learning for Action and Gesture Recognition in Image Sequences: A Survey

“…Caffe has been also used to implement 3D-CNN for action recognition (Tran et al, 2015;Poleg et al, 2016;Shou et al, 2016b;Wang et al, 2016d;Singh et al, 2016b), and motionbased approaches for both action (Simonyan and Zisserman, 2014;Singh et al, 2016a;Gkioxari and Malik, 2015) and gesture recognition (Wu et al, 2016b;Wang et al, 2017. Caffe is preferred to other frameworks for its speed and efficiency, especially in "fused" architectures for action recognition (Singh et al, 2016b;Deng et al, 2015;Diba et al, 2016;Peng and Schmid, 2016). Popular network types like FNN, CNN, LSTM, and RNN are fully supported by CNTK (Yu et al, 2014), which was started by speech processing researchers.…”