Self-Supervised Pretraining Improves Self-Supervised Pretraining

Reed, Colorado; Yue, Xiangyu; Nrusimha, Ani; Ebrahimi, Sayna; Vijaykumar, V.; Mao, Richard; Li, Bo; Zhang, Shanghang; Guillory, Devin; Metzger, Sean; Keutzer, Kurt; Darrell, Trevor

doi:10.48550/arxiv.2103.12718

Cited by 6 publications

(6 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recent studies in pre-training for remote sensing usually involve some pre-text tasks like the colorization of images [42], super-resolution of imagery [43], or classifying whether two patches overlap [44]. It is also possible to prepre-train a network on natural imagery before pre-training on aerial imagery in a second step [45].…”

Section: Pre-trained Models In Remote Sensingmentioning

confidence: 99%

Self-supervised audiovisual representation learning for remote sensing data

Heidler

Mou

et al. 2023

International Journal of Applied Earth Observation and Geoinfor

View full text Add to dashboard Cite

Section: Pre-trained Models In Remote Sensingmentioning

confidence: 99%

Self-supervised audiovisual representation learning for remote sensing data

Heidler

Mou

et al. 2023

International Journal of Applied Earth Observation and Geoinfor

View full text Add to dashboard Cite

“…In this sense, Foundation Models training is an approach that trades off the need for task-specific data with the need for large amounts of data at pretraining. This is leveraged in hierarchical self-supervised pretraining which consists of a sequence of self-supervised training steps on decreasing amounts of increasingly task-relevant data, so as to tune the trade-off between data quantity and quality in ways that best match the data availability [10], [11].…”

Section: Data Requirementsmentioning

confidence: 99%

Towards Workflows for the Use of AI Foundation Models in Visual Inspection Applications

Rigotti,

Antognini,

Assaf

et al. 2023

ce papers

View full text Add to dashboard Cite

The latest successes in AI have been largely driven by a paradigm known as Foundation Models (FMs), large Neural Networks pretrained on massive datasets that thereby acquire impressive transfer learning capabilities to adapt to new tasks. The emerging properties of FMs have unlocked novel tantalizing applications for instance enabling the generation of fluent text and realistic images from text descriptions. The impact of FMs on technical domains like civil engineering is however still in its infancy, owing to a gap between research development and application use cases. This paper aims to help bridge this gap and promote adoption among technical practitioners, specifically in visual inspection applications for civil engineering. For that we analyze the requirements in terms of data availability making particular use cases amenable to the pretraining/fine‐tuning paradigm of FMs, i.e. situations where labeled data is scarce or costly, but unlabeled data is abundant. We then illustrate proof‐of‐concepts workflows using FMs, in visual inspection applications. We hope that our contribution will mark the start of conversations between AI researchers and civil engineers on the potential of FMs to accelerate workflows supporting vision tasks for maintenance inspections and decisions.

show abstract

“…Self-supervised learning often involves various image restoration (e.g., inpainting [40], colorization [58], denoising [48]) and higher level prediction tasks like image orientation [19], context [14], temporal ordering [38], and cluster assignments [5]. The learned representations transfer well to image classification but the improvement is less significant for instance-level tasks, such as object detection and instance segmentation [24,41].…”

Section: Related Workmentioning

confidence: 99%

DETReg: Unsupervised Pretraining with Region Priors for Object Detection

Bar¹,

Wang²,

Kantorov³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Self-supervised pretraining has recently proven beneficial for computer vision tasks, including object detection. However, previous self-supervised approaches are not designed to handle a key aspect of detection: localizing objects. Here, we present DETReg, an unsupervised pretraining approach for object DEtection with TRansformers using Region priors. Motivated by the two tasks underlying object detection: localization and categorization, we combine two complementary signals for self-supervision. For an object localization signal, we use pseudo ground truth object bounding boxes from an off-the-shelf unsupervised region proposal method, Selective Search, which does not require training and can detect objects at a high recall rate and very low precision. The categorization signal comes from an object embedding loss that encourages invariant object representations, from which the object category can be inferred. We show how to combine these two signals to train the Deformable DETR detection architecture from large amounts of unlabeled data. DETReg improves the performance over competitive baselines and previous self-supervised methods on standard benchmarks like MS COCO and PASCAL VOC. DETReg also outperforms previous supervised and unsupervised baseline approaches for a low-data regime when trained with only 1%, 2%, 5%, and 10% of the labeled data on MS COCO. For code and pretrained models, visit the project page https://amirbar.net/detreg.Preprint. Under review.

show abstract

Self-Supervised Pretraining Improves Self-Supervised Pretraining

Cited by 6 publications

References 37 publications

Self-supervised audiovisual representation learning for remote sensing data

Self-supervised audiovisual representation learning for remote sensing data

Towards Workflows for the Use of AI Foundation Models in Visual Inspection Applications

DETReg: Unsupervised Pretraining with Region Priors for Object Detection

Contact Info

Product

Resources

About