W-TALC: Weakly-Supervised Temporal Activity Localization and Classification

Abstract

Most activity localization methods in the literature suffer from the burden of frame-wise annotation requirement. Learning from weak labels may be a potential solution towards reducing such manual labeling effort. Recent years have witnessed a substantial influx of tagged videos on the Internet, which can serve as a rich source of weakly-supervised training data. Specifically, the correlations between videos with similar tags can be utilized to temporally localize the activities. Towards this goal, we present W-TALC, a Weakly-supervised Temporal Activity Localization and Classification framework using only video-level labels. The proposed network can be divided into two sub-networks, namely the Two-Stream based feature extractor network and a weakly-supervised module, which we learn by optimizing two complimentary loss functions. Qualitative and quantitative results on two challenging datasets -Thumos14 and ActivityNet1.2, demonstrate that the proposed method is able to detect activities at a fine granularity and achieve better performance than current state-of-the-art methods. Codes available at https://github.com/sujoyp/wtalc-pytorch

3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization

“…We then present our overall architecture followed by a detailed description of the different loss terms in the proposed formulation. Feature Extraction: As in [14,16], we use Inflated 3D (I3D) features extracted from the RGB and flow I3D deep networks [4], trained on the Kinetics dataset, to encode appearance and motion information, respectively. A video is divided into non-overlapping segments, each consisting of 16 frames.…”

“…This leads to improved action localization results. However, the formulation in [16] puts a constraint on the mini-batch, used for training, to mostly contain paired videos with actions belonging to the same category. In this work, we look into an alternative formulation that allows the mini-batch to contain diverse action samples during training.…”

“…We propose a framework, called 3C-Net, using a novel formulation to learn discriminative action features with enhanced localization capabilities using video-level supervision. As in [14,16], our formulation contains a classification loss term that ensures the inter-class separability of learned features, for video-level action classification. However, this separability at the global video-level alone is insufficient for accurate action localization, which is generally a local temporal-context classification.…”

“…Our approach sets a new state-of-the-art for weakly-supervised temporal action localization on both datasets. On the THU-MOS14 dataset, the proposed method achieves an absolute gain of 4.6% in terms of mean average precision (mAP), compared to the state-of-the-art [16]. Source code is available at https://github.com/naraysa/3c-net.…”

AdapNet: Adaptability Decomposing Encoder–Decoder Network for Weakly Supervised Action Recognition and Localization

“…AutoLoc [31] is proposed to directly predict the temporal boundary of each action instance with an outer-inner-contrastive loss to train the boundary predictor. W-TALC [32] learns the specific network weights by optimizing two complimentary loss functions, namely coactivity similarity loss and multiple instance learning loss. To fully leverage the publicly available trimmed videos, this paper further studies a reliable knowledge transfer mechanism from trimmed to untrimmed videos under adaptability constraint for effective action recognition and localization with weak supervision.…”