Learn to cycle: Time-consistent feature discovery for action recognition

Stergiou, Alexandros; Poppe, Ronald

doi:10.1016/j.patrec.2020.11.012

Cited by 22 publications

(15 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For MTNet M (CR), we note a performance increase of +1.3% in the top-1 and +0.8% in the top-5 compared to the original network without CR. These accuracy rates are similar to both of the larger SRTG r3d-101 [243] and Channel-Separated Networks (ip-CSN-101) [260]. This increase in performance does not come at a cost in terms of the computational inference or the number of parameters with the additional number of parameters accounting for ∼ 10% of the total number of network parameters.…”

Section: Results On Hacssupporting

confidence: 61%

“…Notably, it also comes at a negligible computational overhead when added over MTNet (< +1%). More specifically, considering the overall efficiency of the MTNets in addition to the proposed methods, our MTNet S with CR can achieve accuracy rates similar to those of ResNet-101 with 3D or ResNet-50 with (2+1)D convolutions [261], as well as TAM [62] and SRTG r3d-50 [243]. However, the GFLOP requirements are very low, similar to the original MTNet S with only an additional > 0.1 GFLOPs that account for less than 10% of the total 5.8 GFLOPs.…”

Section: Results On Hacsmentioning

confidence: 99%

“…We include wide Residual Networks in our tests in order to have a greater understanding of the effects of larger feature dimension sizes. We do not explicitly include experiments with SR [243] networks from Chapter 4, as MTNets include SR as part of their architectures.…”

Section: Methodsmentioning

confidence: 99%

“…The concatenated activations of the local (L) and prolonged (P) branches are aligned based on the importance of each feature in the context of the entire video sequence. The role of the global aggregated feature importance branch (G) is the creation of coherent activations based on averaged feature attention through Squeeze and Recursion [243] with GRU [40] recurrent cells. The branch operates over a vectorised version of the original volume pooled by its spatial dimensionality (pool(a [l−1] )).…”

Section: Mtblocks Structurementioning

confidence: 99%

See 3 more Smart Citations

Efficient Modelling Across Time of Human Actions and Interactions

Stergiou¹

View full text Add to dashboard Cite

This thesis focuses on video understanding for human action and interaction recognition. We start by identifying the main challenges related to action recognition from videos and review how they have been addressed by current methods.Based on these challenges, and by focusing on the temporal aspect of actions, we argue that current fixed-sized spatio-temporal kernels in 3D convolutional neural networks (CNNs) can be improved to better deal with temporal variations in the input. Our contributions are based on the enlargement of the convolutional receptive fields through the introduction of spatio-temporal size-varying segments of videos, as well as the discovery of the local feature relevance over the entire video sequence. The resulting extracted features encapsulate information that includes the importance of local features across multiple temporal durations, as well as the entire video sequence.Subsequently, we study how we can better handle variations between classes of actions, by enhancing their feature differences over different layers of the architecture. The hierarchical extraction of features models variations of relatively similar classes the same as very dissimilar classes. Therefore, distinctions between similar classes are less likely to be modelled. The proposed approach regularises feature maps by amplifying features that correspond to the class of the video that is processed. We move away from class-agnostic networks and make early predictions based on feature amplification mechanism.The proposed approaches are evaluated on several benchmark action recognition datasets and show competitive results. In terms of performance, we compete with the state-of-the-art while being more efficient in terms of GFLOPs.Finally, we present a human-understandable approach aimed at providing visual explanations for features learned over spatio-temporal networks. We isolate spatio-temporal regions in 3D-CNNs that are informative for an action class. We extend this approach to allow for the traversal over the entire network architecture, incrementally discovering kernels at different complexities, and modelling layers related to a specific class.ix Chapter 1 IntroductionChapter 2, Related Work. We detail a synopsis of current progress made in action recognition. We distinguish between approaches that have been based on hand-crafted features and methods that used learned features through optimisation. We then present the main motivation for the works that have been included in this thesis. We discuss how our approaches differe from previous efforts and the challenges that our methods address.Chapter 3, Datasets for Video Understanding. We provide an overview of historic and current datasets used as action recognition benchmarks. We focus on datasets that exemplify milestones achieved in terms of data collection and increases in the data complexity. We then present the datasets that are used in this thesis.Chapter 4, Improving Action Recognition through Time-Consistent Features. We present a novel method to address var...

show abstract

Section: Results On Hacssupporting

confidence: 61%

Section: Results On Hacsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Mtblocks Structurementioning

confidence: 99%

See 2 more Smart Citations

Efficient Modelling Across Time of Human Actions and Interactions

Stergiou¹

View full text Add to dashboard Cite

show abstract

“…Others have worked on the minimization of the computational requirements by shifting activations along time [25]. Previous works that have touched upon regularization were aimed more at the overall temporal consistency of features (e.g., [26]) without the consideration of feature combinations that are most informative for some classes.…”

Section: Related Workmentioning

confidence: 99%

Learning Class-Specific Features with Class Regularization for Videos

2020

Self Cite

View full text Add to dashboard Cite

One of the main principles of Deep Convolutional Neural Networks (CNNs) is the extraction of useful features through a hierarchy of kernels operations. The kernels are not explicitly tailored to address specific target classes but are rather optimized as general feature extractors. Distinction between classes is typically left until the very last fully-connected layers. Consequently, variances between classes that are relatively similar are treated the same way as variations between classes that exhibit great dissimilarities. In order to directly address this problem, we introduce Class Regularization, a novel method that can regularize feature map activations based on the classes of the examples used. Essentially, we amplify or suppress activations based on an educated guess of the given class. We can apply this step to each minibatch of activation maps, at different depths in the network. We demonstrate that this improves feature search during training, leading to systematic improvement gains on the Kinetics, UCF-101, and HMDB-51 datasets. Moreover, Class Regularization establishes an explicit correlation between features and class, which makes it a perfect tool to visualize class-specific features at various network depths.

show abstract

Linear dynamical systems approach for human action recognition with dual-stream deep features

Mukaidani

2021

Appl Intell

View full text Add to dashboard Cite

Learn to cycle: Time-consistent feature discovery for action recognition

Cited by 22 publications

References 5 publications

Efficient Modelling Across Time of Human Actions and Interactions

Efficient Modelling Across Time of Human Actions and Interactions

Learning Class-Specific Features with Class Regularization for Videos

Linear dynamical systems approach for human action recognition with dual-stream deep features

Contact Info

Product

Resources

About