AutoLoc: Weakly-Supervised Temporal Action Localization in Untrimmed Videos

Abstract

Temporal Action Localization (TAL) in untrimmed video is important for many applications. But it is very expensive to annotate the segment-level ground truth (action class and temporal boundary). This raises the interest of addressing TAL with weak supervision, namely only video-level annotations are available during training). However, the state-of-the-art weakly-supervised TAL methods only focus on generating good Class Activation Sequence (CAS) over time but conduct simple thresholding on CAS to localize actions. In this paper, we first develop a novel weakly-supervised TAL framework called AutoLoc to directly predict the temporal boundary of each action instance. We propose a novel Outer-Inner-Contrastive (OIC) loss to automatically discover the needed segment-level supervision for training such a boundary predictor. Our method achieves dramatically improved performance: under the IoU threshold 0.5, our method improves mAP on THUMOS'14 from 13.7% to 21.2% and mAP on ActivityNet from 7.4% to 27.3%. It is also very encouraging to see that our weakly-supervised method achieves comparable results with some fully-supervised methods.

3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization

“…There are an average of 1.5 activity instances per video. As in [22,16], we use the training set to train and validation set to test our approach. Count Labels: The ground-truth count labels for the videos in both datasets are generated using the available temporal action segments information.…”

“…We report mAP scores at different IoU thresholds. Both UntrimmedNets [28] and Autoloc [22] use TSN [29] as the backbone, whereas STPN [14] and W-TALC [16] use I3D networks similar to our framework. The STPN approach obtains an mAP of 16.9 at IoU=0.5, while W-TALC achieves an mAP of 22.0.…”

AdapNet: Adaptability Decomposing Encoder–Decoder Network for Weakly Supervised Action Recognition and Localization

“…TSR-Net [30] integrates self-attention and transfer learning with temporal localization framework to obtain precise temporal intervals in untrimmed videos. AutoLoc [31] is proposed to directly predict the temporal boundary of each action instance with an outer-inner-contrastive loss to train the boundary predictor. W-TALC [32] learns the specific network weights by optimizing two complimentary loss functions, namely coactivity similarity loss and multiple instance learning loss.…”

WSLLN:Weakly Supervised Natural Language Localization Networks

“…Weakly Supervised Localization has been studied extensively to use weak supervisions for object detection on images and action localization in videos (Oquab et al, 2015;Bilen and Vedaldi, 2016;Kantorov et al, 2016;Jie et al, 2017;Diba et al, 2017;Papadopoulos et al, 2017;Duchenne et al, 2009;Laptev et al, 2008;Bojanowski et al, 2014;Shou et al, 2018a). Some methods use class labels to train object detectors.…”

“…Gao et al utilized object counts for weakly supervised object detection . Instead of using temporally labeled segments, weakly supervised action detectors use weaker annotations, e.g., movie script (Duchenne et al, 2009;Laptev et al, 2008), the order of the occurring action classes in videos (Bojanowski et al, 2014; and video-level class labels Shou et al, 2018a).…”