AASAE: Augmentation-Augmented Stochastic Autoencoders

Falcon, William; Jha, Ananya Harsh; Koker, Teddy; Cho, Kyunghyun

doi:10.48550/arxiv.2107.12329

Cited by 3 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we do not consider methods based on VAEs [1,45], since they have been shown to yield poor performance in the large and medium scale. For instance, as found by [21], a VAE trained offline on CIFAR10 reaches an accuracy of 57.2%, which is lower than any method (except VICReg) trained continually on CIFAR100 with CaSSLe.…”

Section: B Derivation Of Distillation Lossesmentioning

confidence: 82%

Self-Supervised Models are Continual Learners

Fini¹,

Costa²,

Alameda-Pineda³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophically reduced in a Continual Learning (CL) scenario where data is presented to the model sequentially. In this paper, we show that self-supervised loss functions can be seamlessly converted into distillation mechanisms for CL by adding a predictor network that maps the current state of the representations to their past state. This enables us to devise a framework for Continual self-supervised visual representation Learning that (i) significantly improves the quality of the learned representations, (ii) is compatible with several state-of-the-art self-supervised objectives, and (iii) needs little to no hyperparameter tuning. We demonstrate the effectiveness of our approach empirically by training six popular self-supervised models in various CL settings.

show abstract

Section: B Derivation Of Distillation Lossesmentioning

confidence: 82%

Self-Supervised Models are Continual Learners

Fini¹,

Costa²,

Alameda-Pineda³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Here we fixed P (R) = P (X ), but better anchor priors can be designed which may further improve performance. Finally, anchoring also presents interesting avenues for theoretical analysis to understand its connection to reconstruction-based representation learning approaches such as (Falcon et al 2021) and (Sinha and Dieng 2021).…”

Section: Conclusion and Discussionmentioning

confidence: 99%

$Δ$-UQ: Accurate Uncertainty Quantification via Anchor Marginalization

Thiagarajan¹

2021

Preprint

View full text Add to dashboard Cite

We present ∆-UQ -a novel, general-purpose uncertainty estimator using the concept of anchoring in predictive models. Anchoring works by first transforming the input into a tuple consisting of an anchor point drawn from a prior distribution, and a combination of the input sample with the anchor using a pretext encoding scheme. This encoding is such that the original input can be perfectly recovered from the tuple -regardless of the choice of the anchor. Therefore, any predictive model should be able to predict the target response from the tuple alone (since it implicitly represents the input). Moreover, by varying the anchors for a fixed sample, we can estimate uncertainty in the prediction even using only a single predictive model. We find this uncertainty is deeply connected to improper sampling of the input data, and inherent noise, enabling us to estimate the total uncertainty in any system. With extensive empirical studies on a variety of use-cases, we demonstrate that ∆-UQ outperforms several competitive baselines. Specifically, we study model fitting, sequential model optimization, model based inversion in the regression setting and out of distribution detection, & calibration under distribution shifts for classification.

show abstract

“…Outside of GANs, variational autoencoders (VAEs) have been adapted to generate more semantically meaningful representations by trading off latent channel capacity and independence constraints with reconstruction accuracy (Higgins et al, 2016), an idea that has also been incorporated into recognition improvements using patch-level bottlenecks (Gupta et al, 2020), which encourage a VAE to focus on useful patterns in images. By incorporating data-augmentation, VAE is also shown to achieve fair discriminative performance (Falcon et al, 2021). Recently, works like MAE (He et al, 2021) and CAE have learned representations by solving masked reconstruction tasks using vision transformers.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning of Structured Representations via Closed-Loop Transcription

Tong¹,

Dai²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper proposes an unsupervised method for learning a unified representation that serves both discriminative and generative purposes. While most existing unsupervised learning approaches focus on a representation for only one of these two goals, we show that a unified representation can enjoy the mutual benefits of having both. Such a representation is attainable by generalizing the recently proposed closed-loop transcription framework, known as CTRL, to the unsupervised setting. This entails solving a constrained maximin game over a rate reduction objective that expands features of all samples while compressing features of augmentations of each sample. Through this process, we see discriminative low-dimensional structures emerge in the resulting representations. Under comparable experimental conditions and network complexities, we demonstrate that these structured representations enable classification performance close to state-of-the-art unsupervised discriminative representations, and conditionally generated image quality significantly higher than that of state-of-the-art unsupervised generative models. Source code can be found at https://github.com/Delay-Xili/uCTRL.

show abstract

AASAE: Augmentation-Augmented Stochastic Autoencoders

Cited by 3 publications

References 29 publications

Self-Supervised Models are Continual Learners

Self-Supervised Models are Continual Learners

$Δ$-UQ: Accurate Uncertainty Quantification via Anchor Marginalization

Unsupervised Learning of Structured Representations via Closed-Loop Transcription

Contact Info

Product

Resources

About