Masked Feature Prediction for Self-Supervised Visual Pre-Training

Abstract

We present Masked Feature Prediction (MaskFeat) for self-supervised pre-training of video models. Our approach first randomly masks out a portion of the input sequence and then predicts the feature of the masked regions. We study five different types of features and find Histograms of Oriented Gradients (HOG), a hand-crafted feature descriptor, works particularly well in terms of both performance and efficiency. We observe that the local contrast normalization in HOG is essential for good results, which is in line with earlier work using HOG for visual recognition. Our approach can learn abundant visual knowledge and drive large-scale Transformer-based models. Without using extra model weights or supervision, MaskFeat pretrained on unlabeled videos achieves unprecedented results of 86.7% with MViT-L on Kinetics-400, 88.3% on Kinetics-600, 80.4% on Kinetics-700, 38.8 mAP on AVA, and 75.0% on SSv2. MaskFeat further generalizes to image input, which can be interpreted as a video with a single frame and obtains competitive results on ImageNet.

Corrupted Image Modeling for Self-Supervised Visual Pre-Training

“…Therefore our method can provide abundant nontrivial image pairs feeding the enhancer. (Wei et al, 2021) proposes to predict hand-crafted image feature descriptors at the masked positions. As MIM is originated in masked language modeling (Devlin et al, 2019), CIM is inspired by (Clark et al, 2020).…”

“…(MIM; Bao et al, 2021), which randomly masks out some input tokens and then recovers the masked content by conditioning on the visible context, is able to learn rich visual representations and shows promising performance on various vision benchmarks (Zhou et al, 2021;He et al, 2021;Xie et al, 2021;Dong et al, 2021;Wei et al, 2021;El-Nouby et al, 2021).…”

“…For the first time, we demonstrate that both ViT and CNN can learn rich visual representations using a unified non-Siamese structure. Moreover, the components of CIM, such as the generator, the image tokenizer (Esser et al, 2021;Yu et al, 2021;Dong et al, 2021), the sampling method (Holtzman et al, 2019), as well as the pre-training objective (Wei et al, 2021) can be further customized and improved.…”

CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval

“…In this work, we found that our proposed MIM is more effective than MLM. Inspired by recent works of self-supervised learning on vision [12,42], we propose to mask out image patches with larger proportion and follow MaskFeat [41] to reconstruct other views of the whole image rather than recovering those masked regions only.…”

“…Our supervision is provided by another view of the original intact input image. While MaskFeat [41] used Histograms of Oriented Gradients (HOG) as the supervision for visual pre-training, we rely on the more discriminative signals from ClusterFit [43] and GrokNet [4] to extract additional two views of the raw image. Between the two, [43] provides the clustering probability 𝑐 (v) while [4] extracts the pool5 embedding 𝑟 (v) (feature output from the 5-th Conv Block).…”