Dance With Flow: Two-In-One Stream Action Detection

Zhao, Jiaojiao; Snoek, Cees G. M.

doi:10.1109/cvpr.2019.01017

Cited by 89 publications

(45 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Resolution of individual frames remains the same (240 × 320) without cropping and resizing, and multiple modalities utilizing 2stream frame -RGB and stacked optical flow are exploited. Proposals of bounding boxes from each stream are merged with non-maximum suppression (NMS)(refer to [24] for more detail).…”

Section: B Setups and Hyper-parametersmentioning

confidence: 99%

Convolution Encoders for End-to-End Action Tracking With Space-Time Cubic Kernels

Wang

Hawbani

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Convolution is good at extracting per-frame representations but discards the temporal order. Most previous Convolution Neural Network (CNN) based video action tracking handle space and time dynamics separately. To remedy this issue, we extend the convolution kernel with a time dimension and formulate the problem to an encoder-decoder structure, which generates the bounding boxes of action regions for each frame by leveraging the pixel-wise annotations. By using the same measure, Cubic Tracker achieves mAP of 64.9% (frame IoU threshold = 0.5), 78.6%(video IoU threshold = 0.2), and 78.3% (video IoU threshold = 0.5) with gains of 3.6%, 0.2%, and 1.4% over the best competitors on J-HMDB. More remarkably, on 2 benchmarks without pixel-level supervision (UCF-Sports and UCF101-24), Cubic Tracker also approaches the performance of the state of the arts, by simply transferring learned model from J-HMDB, with no further adaptive fine-tuning techniques applied.

show abstract

Section: B Setups and Hyper-parametersmentioning

confidence: 99%

Convolution Encoders for End-to-End Action Tracking With Space-Time Cubic Kernels

Wang

Hawbani

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Such motion-based attention can be especially useful for the intended setting utilizing stationary cameras. Building on this technique of motion-based attention, Zhao and Snoek developed an algorithm for detecting the spatiotemporal extent of actions by embedding the RGB spatial and optical flow temporal streams into a single two-in-one stream network [101]. Aside from simplifying the computation of action recognition, their approach also assigns motion direction to the actor as an extra feature distinctive to many actions (e.g., the difference between sitting down and standing up, or PA-entailing motion towards a direction relevant to the research questions studied with proposed smart sensors).…”

Section: The Promise Of Computer Visionmentioning

confidence: 99%

Computer Vision-Based Unobtrusive Physical Activity Monitoring in School by Room-Level Physical Activity Estimation: A Method Proposition

Hõrak

2019

Information

View full text Add to dashboard Cite

As sedentary lifestyles and childhood obesity are becoming more prevalent, research in the field of physical activity (PA) has gained much momentum. Monitoring the PA of children and adolescents is crucial for ascertaining and understanding the phenomena that facilitate and hinder PA in order to develop effective interventions for promoting physically active habits. Popular individual-level measures are sensitive to social desirability bias and subject reactivity. Intrusiveness of these methods, especially when studying children, also limits the possible duration of monitoring and assumes strict submission to human research ethics requirements and vigilance in personal data protection. Meanwhile, growth in computational capacity has enabled computer vision researchers to successfully use deep learning algorithms for real-time behaviour analysis such as action recognition. This work analyzes the weaknesses of existing methods used in PA research; gives an overview of relevant advances in video-based action recognition methods; and proposes the outline of a novel action intensity classifier utilizing sensor-supervised learning for estimating ambient PA. The proposed method, if applied as a distributed privacy-preserving sensor system, is argued to be useful for monitoring the spatio-temporal distribution of PA in schools over long periods and assessing the efficiency of school-based PA interventions.

show abstract

“…Following architectures have focused on modelling longer temporal structure, through consensus of predictions over time [30,58,63] as well as inflating CNNs to 3D convolutions [6], all using the two-stream approach of latefusing RGB and Flow. The latest architectures have focused on reducing the high computational cost of 3D convolutions [12,21,61], yet still show improvements when reporting results of two-stream fusion [61]. Self-supervision for Action Recognition.…”

Section: Related Workmentioning

confidence: 99%

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

Munro

Damen

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Fine-grained action recognition datasets exhibit environmental bias, where multiple video sequences are captured from a limited number of environments. Training a model in one environment and deploying in another results in a drop in performance due to an unavoidable domain shift. Unsupervised Domain Adaptation (UDA) approaches have frequently utilised adversarial training between the source and target domains. However, these approaches have not explored the multi-modal nature of video within each domain. In this work we exploit the correspondence of modalities as a self-supervised alignment approach for UDA in addition to adversarial alignment (Fig. 1).We test our approach on three kitchens from our largescale dataset, EPIC-Kitchens [8], using two modalities commonly employed for action recognition: RGB and Optical Flow. We show that multi-modal self-supervision alone improves the performance over source-only training by 2.4% on average. We then combine adversarial training with multi-modal self-supervision, showing that our approach outperforms other UDA methods by 3%.

show abstract

Dance With Flow: Two-In-One Stream Action Detection

Cited by 89 publications

References 45 publications

Convolution Encoders for End-to-End Action Tracking With Space-Time Cubic Kernels

Convolution Encoders for End-to-End Action Tracking With Space-Time Cubic Kernels

Computer Vision-Based Unobtrusive Physical Activity Monitoring in School by Room-Level Physical Activity Estimation: A Method Proposition

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

Contact Info

Product

Resources

About