Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition

Abstract

Current fully-supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain-specific labels. This hinders the progress towards advanced video architectures. This paper presents an in-depth study of using large volumes of web videos for pre-training video models for the task of action recognition. Our primary empirical finding is that pre-training at a very large scale (over 65 million videos), despite on noisy social-media videos and hashtags, substantially improves the state-of-the-art on three challenging public action recognition datasets. Further, we examine three questions in the construction of weakly-supervised video action datasets. First, given that actions involve interactions with objects, how should one construct a verb-object pretraining label space to benefit transfer learning the most? Second, frame-based models perform quite well on action recognition; is pre-training for good image features sufficient or is pre-training for spatio-temporal features valuable for optimal transfer learning? Finally, actions are generally less well-localized in long videos vs. short videos; since action labels are provided at a video level, how should one choose video clips for best performance, given some fixed budget of number or minutes of videos?

Transferability and Hardness of Supervised Classification Tasks

“…Analysis of results. Subject specific attributes such as male and bald are evidently more transferable from recognition (left columns of Table 1) than attributes that are related to Although this relationship has been noted by others, previous work used domain knowledge to determine which attributes are more transferable from identity [35], as others have done in other domains [20,38]. By comparison, our work shows how these relationships emerge from our estimation of transferability.…”

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

“…Results and Discussion. Our results are presented in Table 6 and compared to state-of-the-art methods [69], [72], [96], [97], [98]. Our final model, using only RGB frames, achieves state-of-the-art results in comparison to all prior work, including those use optical flow [72], object detector [98] or audio data [69].…”

Video Classification With Channel-Separated Convolutional Networks

“…Our ip-CSN-152 is still 0.6% lower than SlowFast augmented with Non-Local Networks. Finally, recent work [13] has shown that R(2+1)D can achieve strong performance when pre-trained on a large-scale weakly-supervised dataset. We pre-train/finetune ir-and ip-CSN-152 on the same dataset and compare it with R(2+1)D-152 (the last three rows of Table 5).…”