Fortuitous Forgetting in Connectionist Networks

Zhou, Hattie; Vani, Ankit; Larochelle, Hugo; Courville, Aaron

doi:10.48550/arxiv.2202.00155

Cited by 2 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Forgetting The use of probabilistic models such as GMMs has another interesting consequence, namely the ability to control forgetting. Following Zhou et al (2022), forgetting can be a beneficial functionality, which is simple to control in GMMs by eliminating certain components without impacting the remaining ones.…”

Section: Ar Assumptionsmentioning

confidence: 99%

Adiabatic replay for continual learning

Krawczyk¹,

Gepperth²

2023

Preprint

View full text Add to dashboard Cite

Conventional replay-based approaches to continual learning (CL) require, for each learning phase with new data, the replay of samples representing all of the previously learned knowledge in order to avoid catastrophic forgetting. Since the amount of learned knowledge grows over time in CL problems, generative replay spends an increasing amount of time just re-learning what is already known. In this proof-of-concept study, we propose a replay-based CL strategy that we term adiabatic replay (AR), which derives its efficiency from the (reasonable) assumption that each new learning phase is adiabatic, i.e., represents only a small addition to existing knowledge. Each new learning phase triggers a sampling process that selectively replays, from the body of existing knowledge, just such samples that are similar to the new data, in contrast to replaying all of it. Complete replay is not required since AR represents the data distribution by GMMs, which are capable of selectively updating their internal representation only where data statistics have changed. As long as additions are adiabatic, the amount of to-be-replayed samples need not to depend on the amount of previously acquired knowledge at all. We verify experimentally that AR is superior to state-of-the-art deep generative replay using VAEs.

show abstract

Section: Ar Assumptionsmentioning

confidence: 99%

Adiabatic replay for continual learning

Krawczyk¹,

Gepperth²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Regularization in Self-supervised Learning. The concept of regularizing a specific subset of the network is relatively unexplored in self-supervised learning but finds motivation in recent findings from supervised settings, such as addressing minority examples (Hooker et al, 2019), out-ofdistribution generalization , late-layer regularizations through head weight-decay (Abnar et al, 2021), and initialization (Zhou et al, 2022). Additionally, Lee et al (2022a) propose surgically fine-tuning specific layers of the network to handle distribution shifts in particular categories.…”

Section: Related Workmentioning

confidence: 99%

Microgrids and Resilience: A Review

Hamidieh

Ghassemi

2022

IEEE Access

View full text Add to dashboard Cite

Supervised learning methods have been found to exhibit inductive biases favoring simpler features. When such features are spuriously correlated with the label, this can result in suboptimal performance on minority subgroups. Despite the growing popularity of methods which learn from unlabeled data, the extent to which these representations encode spurious features is unclear. In this work, we explore the impact of spurious features on Self-Supervised Learning (SSL) for visual representation learning. We first empirically show that commonly used augmentations in SSL can cause undesired invariances in the image space, and illustrate this with a simple example. We further show that classical approaches in combating spurious correlations, such as dataset re-sampling during SSL, do not consistently lead to invariant representations. Motivated by these findings, we propose LATETVG to remove spurious information from these representations during pretraining, by regularizing later layers of the encoder via pruning. We find that our method produces representations which outperform the baselines on several benchmarks, without the need for group or label information during SSL.

show abstract

Fortuitous Forgetting in Connectionist Networks

Cited by 2 publications

References 34 publications

Adiabatic replay for continual learning

Adiabatic replay for continual learning

Microgrids and Resilience: A Review

Contact Info

Product

Resources

About