Gradient based sample selection for online continual learning

Abstract

A continual learning agent learns online with a non-stationary and never-ending stream of data. The key to such learning process is to overcome the catastrophic forgetting of previously seen data, which is a well known problem of neural networks. To prevent forgetting, a replay buffer is usually employed to store the previous data for the purpose of rehearsal. Previous works often depend on task boundary and i.i.d. assumptions to properly select samples for the replay buffer. In this work, we formulate sample selection as a constraint reduction problem based on the constrained optimization view of continual learning. The goal is to select a fixed subset of constraints that best approximate the feasible region defined by the original constraints. We show that it is equivalent to maximizing the diversity of samples in the replay buffer with parameters gradient as the feature. We further develop a greedy alternative that is cheap and efficient. The advantage of the proposed method is demonstrated by comparing to other alternatives under the continual learning setting. Further comparisons are made against state of the art methods that rely on task boundaries which show comparable or even better results for our method.

New Insights on Reducing Abrupt Representation Change in Online Continual Learning

“…As such, most of the solutions to the online continual learning problem rely on the use of a buffer formed of previous memories, these memories are retrieved when learning new samples. A good body of the online continual learning works (Borsos et al, 2020;Chaudhry et al, 2019;Aljundi et al, 2019) proposes solutions to select which samples should be stored. Alternatively, (Caccia et al, 2019) investigate which samples should be retrieved.…”

“…Anytime evaluation and computational constraints A critical component of continual learning systems particularly in the "online" setting is the ability to use the learner at any point (De Lange et al, 2019). Although most works in the online (one-pass through the data) setting report results (Lopez-Paz et al, 2017;Aljundi et al, 2019) throughout the stream, several prior works have reported the final accuracy as a proxy (Caccia et al, 2019;Shim et al, 2020). However a lack of anytime evaluation opens the possibility to exploit the metrics by proposing offline learning baselines which are inherently not compatible with anytime evaluation (Prabhu et al, 2020) and are not "online" learners.…”

“…This allows us to fairly compare different approaches, as these parameters have a direct impact on the computational cost of a given run. The model architecture is also kept constant, which is a reduced ResNet-18 used in (Lopez-Paz et al, 2017;Caccia et al, 2019;Aljundi et al, 2019), where the dimensions of the last linear layer change depending on the input height and width.…”

“…In this online continual learning setting, to prevent forgetting, methods usually rely on storing a small buffer of previous training data and replaying samples from it as new data is learned. Various works focus on studying which samples to store (Borsos et al, 2020;Aljundi et al, 2019) or which samples to replay when receiving new data (Caccia et al, 2019). In this work, we direct our attention to the representation being learned and investigate how the features of previously learned classes change and drift over time.…”

Overcome Anterograde Forgetting with Cycled Memory Networks

“…The majority of previous works focused on addressing the well-known catastrophic forgetting issue (Mc-Closkey and Cohen 1989). According to the mechanism of memory consolidation, current approaches are categorized into three types: (i) Experiential rehearsal-based approaches, which focus on replaying episodic memory (Robins 1995), and the core of which is to select representative samples or features from historical data (Rebuffi et al 2017;Aljundi et al 2019;Bang et al 2021). (ii) Distributed memory representation approaches (Fernando et al 2017;Mallya and Lazebnik 2018), which allocate individual networks for specific knowledge to avoid interference, represented by Progressive Neural Networks (PNN) (Rusu et al 2016).…”