Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning

Luo, Zhekun; Guillory, Devin; Shi, Baifeng; Ke, Wei; Wan, Fang; Darrell, Trevor; Xu, Huijuan

doi:10.1007/978-3-030-58526-6_43

Cited by 92 publications

(47 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This again demonstrates the superiority of the proposed feature enhancement network. (Paul, Roy, and Roy-Chowdhury 2018) 37.0 12.7 1.5 18.0 CleanNet 37.1 20.3 5.0 21.6 Liu et al (Liu, Jiang, and Wang 2019) 36.8 22.9 5.6 22.4 TSM (Yu et al 2019) 28.3 17.0 3.5 17.1 RPN 37.6 23.9 5.4 23.3 Gong et al (Gong et al 2020) 40.0 25.0 4.6 24.6 EM-MIL (Luo et al 2020) 37.4 --20.3 TSCN (Zhai et al 2020) 37.6 23.7 5.7 23.6 BaS-Net (Lee, Uh, and Byun 2020) 38.5 24.2 5.6 24.3 BaS-Net* (Lee, Uh, and Byun 2020) 36.9 23.3 5.1 22.4 +ACGNet 40.8 +3.9 25.3 +2.0 5.6 +0.5 25.1 +2.7 DGAM (Shi et al 2020) 41.0 23.5 5.3 24.4 DGAM* (Shi et al 2020) 40.3 23.2 5.0 24.0 +ACGNet 41.4 +1.1 24.2 +1.0 5.5 +0.5 24.9 +0.9 BaM (Lee et al 2021) 41.2 25.6 6.0 25.9 BaM* (Lee et al 2021) 40.8 24.9 5.8 25.6 +ACGNet 41.8 +1.0 26.0 +1.1 5.9 +0.1 26.1 +0.5…”

Section: Comparison To State-of-the-art Methodsmentioning

confidence: 99%

“…For example, P-GCN (Zeng et al 2019) constructs a graph according to the distances and IoUs between proposals, aiming to adjust the category and boundary of each proposal by using context information. G-TAD (Xu et al 2020) attempts to make use of not only temporal context, but also semantic context captured through graph convolutional networks (GCN), and then temporal action detection is cast as a sub-graph localization problem. GTRM (Huang, Sugano, and Sato 2020) employs GCN to integrate all the action segments within a certain period of time in the action segmentation task.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

ACGNet: Action Complement Graph Network for Weakly-supervised Temporal Action Localization

Yang¹,

Qin²,

Huang³

2021

Preprint

View full text Add to dashboard Cite

Weakly-supervised temporal action localization (WTAL) in untrimmed videos has emerged as a practical but challenging task since only video-level labels are available. Existing approaches typically leverage off-the-shelf segment-level features, which suffer from spatial incompleteness and temporal incoherence, thus limiting their performance. In this paper, we tackle this problem from a new perspective by enhancing segment-level representations with a simple yet effective graph convolutional network, namely action complement graph network (ACGNet). It facilitates the current video segment to perceive spatial-temporal dependencies from others that potentially convey complementary clues, implicitly mitigating the negative effects caused by the two issues above. By this means, the segment-level features are more discriminative and robust to spatial-temporal variations, contributing to higher localization accuracies. More importantly, the proposed ACGNet works as a universal module that can be flexibly plugged into different WTAL frameworks, while maintaining the end-to-end training fashion. Extensive experiments are conducted on the THUMOS'14 and ActivityNet1.2 benchmarks, where the state-of-the-art results clearly demonstrate the superiority of the proposed approach.

show abstract

Section: Comparison To State-of-the-art Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

ACGNet: Action Complement Graph Network for Weakly-supervised Temporal Action Localization

Yang¹,

Qin²,

Huang³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Weakly supervised temporal action localization provides an efficient way to detect the action instances without overload annotations. Many works mainly tackle this problem using the multiple-instance learning (MIL) framework [12,13,19,20,24,25,32]. Several works [13,35] mainly aggregate snippet-level class scores to produce video-level predictions and learn from video-level action labels.…”

Section: Related Workmentioning

confidence: 99%

“…However, these fully supervised methods require extensive manual frame/snippet level annotations. To address this problem, many weakly supervised temporal action localization (WS-TAL) methods [13,14,25,38,50] are proposed to explore an efficient way to detect the action instances in the given videos with only video-level supervision which is more easily obtained by the annotator.…”

Section: Introductionmentioning

confidence: 99%

Cross-modal Consensus Network for Weakly Supervised Temporal Action Localization

Hong

Feng

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Weakly supervised temporal action localization (WS-TAL) is a challenging task that aims to localize action instances in the given video with video-level categorical supervision. Previous works use the appearance and motion features extracted from pre-trained feature encoder directly, e.g., feature concatenation or score-level fusion. In this work, we argue that the features extracted from the pre-trained extractors, e.g., I3D, which are trained for trimmed video action classification, but not specific for WS-TAL task, leading to inevitable redundancy and sub-optimization . Therefore, the feature re-calibration is needed for reducing the task-irrelevant information redundancy. Here, we propose a cross-modal consensus network (CO 2 -Net) to tackle this problem. In CO 2 -Net, we mainly introduce two identical proposed cross-modal consensus modules (CCM) that design a cross-modal attention mechanism to filter out the task-irrelevant information redundancy using the global information from the main modality and the cross-modal local information from the auxiliary modality. Moreover, we further explore inter-modality consistency, where we treat the attention weights derived from each CCM as the pseudo targets of the attention weights derived from another CCM to maintain the consistency between the predictions derived from two CCMs, forming a mutual learning manner. Finally, we conduct extensive experiments on two commonly used temporal action localization datasets, THUMOS14 and ActivityNet1.2, to verify our method, which we achieve the stateof-the-art results. The experimental results show that our proposed cross-modal consensus module can produce more representative features for temporal action localization.

show abstract

“…A variety of applications are possible using this approach. Recent applications include a progressive learning framework for weakly supervised object detection [25], localization of action segments in video [26], and algorithms within the fields of geoscience and remote sensing [27], [28]. The approach has also been adopted in medical imaging.…”

Section: Related Workmentioning

confidence: 99%

Learning to Localize Objects Using Limited Annotation, With Applications to Thoracic Diseases

2021

View full text Add to dashboard Cite

The localization of objects in images is a longstanding objective within the field of image processing. Most current techniques are based on machine learning approaches, which typically require careful annotation of training samples in the form of expensive bounding box labels. The need for such large-scale annotation has only been exacerbated by the widespread adoption of deep learning techniques within the image processing community: deep learning is notoriously data-hungry. Method: In this work, we attack this problem directly by providing a new method for learning to localize objects with limited annotation: most training images can simply be annotated with their whole image labels (and no bounding box), with only a small fraction marked with bounding boxes. The training is driven by a novel loss function, which is a continuous relaxation of a well-defined discrete formulation of weakly supervised learning. Care is taken to ensure that the loss is numerically well-posed. Additionally, we propose a neural network architecture which accounts for both patch dependence, through the use of Conditional Random Field layers, and shift-invariance, through the inclusion of anti-aliasing filters. Results: We demonstrate our method on the task of localizing thoracic diseases in chest X-ray images, achieving state-of-the-art performance on the ChestX-ray14 dataset. We further show that with a modicum of additional effort our technique can be extended from object localization to object detection, attaining high quality results on the Kaggle RSNA Pneumonia Detection Challenge. Conclusion: The technique presented in this paper has the potential to enable high accuracy localization in regimes in which annotated data is either scarce or expensive to acquire. Future work will focus on applying the ideas presented in this paper to the realm of semantic segmentation.

show abstract

Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning

Cited by 92 publications

References 28 publications

ACGNet: Action Complement Graph Network for Weakly-supervised Temporal Action Localization

ACGNet: Action Complement Graph Network for Weakly-supervised Temporal Action Localization

Cross-modal Consensus Network for Weakly Supervised Temporal Action Localization

Learning to Localize Objects Using Limited Annotation, With Applications to Thoracic Diseases

Contact Info

Product

Resources

About