Imitating the oracle: Towards calibrated model for class incremental learning

Mixture Uniform Distribution Modeling and Asymmetric Mix Distillation for Class Incremental Learning

“…The training data distribution deviates significantly from the target test distribution with growing incremental tasks, which motivates us to explore a novel data distribution model in CIL from the discrepancy perspective. Moreover, inspired by the mixup (Zhang et al 2018(Zhang et al , 2021, recent research communities also applied this training strategy in CIL (Mi et al 2020;Bang et al 2021;Zhu et al 2021;Zhou et al 2022). We extend our modeling framework to mixup and find that discrepancy remains as in Corollary 1 and Fig.…”

Dimensionality reduction for visualizing high-dimensional biological data

“…While this technique aids in the retention of old information, it can be further improved by leveraging the knowledge of the distribution of classes in the feature space. Accordingly, IL2A [56] proposed storing covariance matrices to retain class variations, but this approach can be memory intensive. SSRE [58] proposed a dynamic structure reorganization strategy to retain and transfer knowledge between tasks along with a prototype selection mechanism that utilizes an up-sampling technique of non-augmented class-mean prototypes.…”

“…As F t θ gets updated continually, the actual feature distributions of old classes drift away from their original distributions. To mitigate this drift we incorporate a feature-level knowledge distillation (L KD ) [57,56] that attempts to align the feature spaces of the current and the previous models.…”

“…We compare NAPA-VQ with the existing state-of-the-art (SOTA) methods in NECIL, including EWC [25], LwF MC [38], MUC [34], PASS [57], IL2A [56], and SSRE [58]. We report the average accuracy and average forgetting of NAPA-VQ against the top three performing methods in Ta-bles 1 and 2.…”

“…[57]. While prototypes of old classes in the deep feature space are a viable alternative to reusing exemplars [57,56], if not properly generated, the class boundaries refined using such prototypes tend to be muddled, causing confusion between the old and new classes, and in turn, leading to catastrophic forgetting. To overcome this limitation, we propose NAPA-VQ: Neighborhood-Aware Prototype Augmentation with Vector Quantization framework.…”