Weakly Supervised Action Selection Learning in Video

Ma, Junwei; Gorti, Satya Krishna; Volkovs, Maksims; Yu, Guangwei

doi:10.1109/cvpr46437.2021.00750

Cited by 48 publications

(39 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To solve that, MIL conducts classification for the whole video, during which it carefully selects some parts of the video with high activation scores as action segments. Typically the selection method can be either k-max pooling [44,34,30,40,25], or attention-based pooling [44,31,46,22]. On top of the basic MIL classifier, previous researchers also explored adding different regularization losses, constraints or temporal property modeling to boost the action detection performance.…”

Section: Weakly-supervised Temporal Action Detectionmentioning

confidence: 99%

“…Shou et al [40] added a contrastive loss to help action boundary prediction. And Ma et al [25] introduced a class-agnostic actionness score, which leverages context to help focus on the parts that contain actions.…”

Section: Weakly-supervised Temporal Action Detectionmentioning

confidence: 99%

“…Following prior work [44,34,30,40,25,24], we divide a long video into T short video clips. Then we use feature encoder E to extract clip level features {x 1 , • • • , x T }, where each x i ∈ R d is the d-dimensional feature vector for i th clip.…”

Section: Multiple Instance Learningmentioning

confidence: 99%

See 2 more Smart Citations

Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions

Li¹,

Li²,

Xu³

2022

Preprint

View full text Add to dashboard Cite

Action understanding has evolved into the era of fine granularity, as most human behaviors in real life have only minor differences. To detect these fine-grained actions accurately in a label-efficient way, we tackle the problem of weakly-supervised fine-grained temporal action detection in videos for the first time. Without the careful design to capture subtle differences between fine-grained actions, previous weaklysupervised models for general action detection cannot perform well in the fine-grained setting. We propose to model actions as the combinations of reusable atomic actions which are automatically discovered from data through self-supervised clustering, in order to capture the commonality and individuality of fine-grained actions. The learnt atomic actions, represented by visual concepts, are further mapped to fine and coarse action labels leveraging the semantic label hierarchy. Our approach constructs a visual representation hierarchy of four levels: clip level, atomic action level, fine action class level and coarse action class level, with supervision at each level. Extensive experiments on two large-scale fine-grained video datasets, FineAction and FineGym, show the benefit of our proposed weakly-supervised model for fine-grained action detection, and it achieves state-of-the-art results.

show abstract

Section: Weakly-supervised Temporal Action Detectionmentioning

confidence: 99%

Section: Weakly-supervised Temporal Action Detectionmentioning

confidence: 99%

See 1 more Smart Citation

Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions

Li¹,

Li²,

Xu³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, learning prototypes to represent different aspects of the action (e.g., the start/end of the action) is a difficult task even with annotation [9,40,41]. Thus, it is not sufficient to compute video similarity only by the global prototypes.…”

Section: Introductionmentioning

confidence: 99%

Compound Prototype Matching for Few-shot Action Recognition

Yang¹,

Huang²,

Satô³

2022

Preprint

View full text Add to dashboard Cite

Few-shot action recognition aims to recognize novel action classes using only a small number of labeled training samples. In this work, we propose a novel approach that first summarizes each video into compound prototypes consisting of a group of global prototypes and a group of focused prototypes, and then compares video similarity based on the prototypes. Each global prototype is encouraged to summarize a specific aspect from the entire video, e.g., the start/evolution of the action. Since no clear annotation is provided for the global prototypes, we use a group of focused prototypes to focus on certain timestamps in the video. We compare video similarity by matching the compound prototypes between the support and query videos. The global prototypes are directly matched to compare videos from the same perspective, e.g., to compare whether two actions start similarly. For the focused prototypes, since actions have various temporal variations in the videos, we apply bipartite matching to allow the comparison actions with different temporal positions and shifts. Experiments demonstrate that our proposed method achieves state-of-the-art results on multiple benchmarks.

show abstract

“…Thanks to the various applications [37,55,58], it has drawn much attention from researchers, leading to the rapid and remarkable progress in the fully-supervised setting (i.e., frame-level labels) [31,51,53,60]. Meanwhile, there appear attempts to reduce the prohibitively expensive cost of annotating individual frames by devising weaklysupervised models with video-level labels [8,36,56,66].…”

Section: Introductionmentioning

confidence: 99%

Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization

Lee¹,

Byun²

2021

Preprint

View full text Add to dashboard Cite

We tackle the problem of localizing temporal intervals of actions with only a single frame label for each action instance for training. Owing to label sparsity, existing work fails to learn action completeness, resulting in fragmentary action predictions. In this paper, we propose a novel framework, where dense pseudo-labels are generated to provide completeness guidance for the model. Concretely, we first select pseudo background points to supplement point-level action labels. Then, by taking the points as seeds, we search for the optimal sequence that is likely to contain complete action instances while agreeing with the seeds. To learn completeness from the obtained sequence, we introduce two novel losses that contrast action instances with background ones in terms of action score and feature similarity, respectively. Experimental results demonstrate that our completeness guidance indeed helps the model to locate complete action instances, leading to large performance gains especially under high IoU thresholds. Moreover, we demonstrate the superiority of our method over existing state-ofthe-art methods on four benchmarks: THUMOS'14, GTEA, BEOID, and ActivityNet. Notably, our method even performs comparably to recent fully-supervised methods, at the 6× cheaper annotation cost. Our code is available at https://github.com/Pilhyeon.

show abstract

Weakly Supervised Action Selection Learning in Video

Cited by 48 publications

References 37 publications

Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions

Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions

Compound Prototype Matching for Few-shot Action Recognition

Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization

Contact Info

Product

Resources

About