Noise-based Selection of Robust Inherited Model for Accurate Continual Learning

Du, Xiaocong; Li, Zheng; Seo, Jae Sun; Liu, Frank; Cao, Yu

doi:10.1109/cvprw50498.2020.00130

Cited by 9 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We test the 9+1 single-head accuracy on each class with different memory budgets (Table 3). Compared with (Du et al 2020) which used memory budget of 2,000 IDD samples with the accuracy <90%, our proposed method achieves 91-96% accuracy in any class with only 1,000 IDD samples and competitive performance even with 500 IDD samples.…”

Section: One-class Learning With the Ood Detectormentioning

confidence: 97%

Gradient-Based Novelty Detection Boosted by Self-Supervised Binary Classification

Sun¹,

Yang²,

Zhang³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Novelty detection aims to automatically identify out-of-distribution (OOD) data, without any prior knowledge of them. It is a critical step in data monitoring, behavior analysis and other applications, helping enable continual learning in the field. Conventional methods of OOD detection perform multi-variate analysis on an ensemble of data or features, and usually resort to the supervision with OOD data to improve the accuracy. In reality, such supervision is impractical as one cannot anticipate the anomalous data. In this paper, we propose a novel, self-supervised approach that does not rely on any pre-defined OOD data: (1) The new method evaluates the Mahalanobis distance of the gradients between the in-distribution and OOD data. (2) It is assisted by a self-supervised binary classifier to guide the label selection to generate the gradients, and maximize the Mahalanobis distance. In the evaluation with multiple datasets, such as CIFAR-10, CIFAR-100, SVHN and TinyImageNet, the proposed approach consistently outperforms state-of-the-art supervised and unsupervised methods in the area under the receiver operating characteristic (AUROC) and area under the precision-recall curve (AUPR) metrics. We further demonstrate that this detector is able to accurately learn one OOD class in continual learning.

show abstract

Section: One-class Learning With the Ood Detectormentioning

confidence: 97%

Gradient-Based Novelty Detection Boosted by Self-Supervised Binary Classification

Sun¹,

Yang²,

Zhang³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…We test the 9+1 single-head accuracy on each class with different memory budgets (Table 4). Compared with (Du et al 2020) which used memory budget of 2,000 IDD samples with the accuracy <90%, our proposed method achieves 91-96% accuracy in any class with only 1,000 IDD samples and competitive performance even with 500 IDD samples.…”

Section: W I T H R E -I N I T I a L I Z A T I O N W I T H O U T R E -...mentioning

confidence: 97%

Gradient-based Novelty Detection Boosted by Self-supervised Binary Classification

Sun¹,

Yang²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Novelty detection aims to automatically identify out-ofdistribution (OOD) data, without any prior knowledge of them. It is a critical step in data monitoring, behavior analysis and other applications, helping enable continual learning in the field. Conventional methods of OOD detection perform multi-variate analysis on an ensemble of data or features, and usually resort to the supervision with OOD data to improve the accuracy. In reality, such supervision is impractical as one cannot anticipate the anomalous data. In this paper, we propose a novel, self-supervised approach that does not rely on any pre-defined OOD data: (1) The new method evaluates the Mahalanobis distance of the gradients between the in-distribution and OOD data. (2) It is assisted by a self-supervised binary classifier to guide the label selection to generate the gradients, and maximize the Mahalanobis distance. In the evaluation with multiple datasets, such as CIFAR-10, CIFAR-100, SVHN and TinyImageNet, the proposed approach consistently outperforms state-of-the-art supervised and unsupervised methods in the area under the receiver operating characteristic (AUROC) and area under the precision-recall curve (AUPR) metrics. We further demonstrate that this detector is able to accurately learn one OOD class in continual learning.

show abstract

“…To address this, in this work, we propose a new metric, model stability, from the loss landscape to help shed light on accuracy under variations and model compression and guide an algorithmic solution that mitigates the loss. The model stability is visualized by the loss landscape and evaluated by the roughness score [16]. A lower roughness score indicates a smoother loss landscape and a more stable model.…”

Section: Q U a N T I Z A T I O N 8 -B I T T E R N A R Y R R A M W R I...mentioning

confidence: 99%

“…Given a trained DNN model, its model stability is an intrinsic property to withstand perturbations, such as variations in model weights and input noise. Model stability of a DNN, i.e., the DNN generalization capability, is directly related to the contour of the loss function [16,[19][20][21]. A flatter contour of the loss function leads to a larger region of acceptable minima, which allows the DNN model to better tolerate variations in both weights and inputs.…”

Section: Dnn Model Stabilitymentioning

confidence: 99%

Exploring Model Stability of Deep Neural Networks for Reliable RRAM-Based In-Memory Acceleration

Krishnan

Sun

et al. 2022

IEEE Trans. Comput.

View full text Add to dashboard Cite

RRAM-based in-memory computing (IMC) effectively accelerates deep neural networks (DNNs). Furthermore, model compression techniques, such as quantization and pruning, are necessary to improve algorithm mapping and hardware performance. However, in the presence of RRAM device variations, low-precision and sparse DNNs suffer from severe post-mapping accuracy loss. To address this, in this work, we investigate a new metric, model stability, from the loss landscape to help shed light on accuracy loss under variations and model compression, which guides an algorithmic solution to maximize model stability and mitigate accuracy loss. Based on statistical data from a CMOS/RRAM 1T1R test chip at 65nm, we characterize wafer-level RRAM variations and develop a cross-layer benchmark tool that incorporates quantization, pruning, device variations, model stability, and IMC architecture parameters to assess post-mapping accuracy and hardware performance. Leveraging this tool, we show that a loss-landscape-based DNN model selection for stability effectively tolerates device variations and achieves a post-mapping accuracy higher than that with 50% lower RRAM variations. Moreover, we quantitatively interpret why model pruning increases the sensitivity to variations, while a lower-precision model has better tolerance to variations. Finally, we propose a novel variation-aware training method to improve model stability, in which there exists the most stable model for the best post-mapping accuracy of compressed DNNs. Experimental evaluation of the method shows up to 19%, 21%, and 11% post-mapping accuracy improvement for our 65nm RRAM device, across various precision and sparsity, on CIFAR-10, CIFAR-100, and SVHN datasets, respectively.

show abstract

Noise-based Selection of Robust Inherited Model for Accurate Continual Learning

Cited by 9 publications

References 14 publications

Gradient-Based Novelty Detection Boosted by Self-Supervised Binary Classification

Gradient-Based Novelty Detection Boosted by Self-Supervised Binary Classification

Gradient-based Novelty Detection Boosted by Self-supervised Binary Classification

Exploring Model Stability of Deep Neural Networks for Reliable RRAM-Based In-Memory Acceleration

Contact Info

Product

Resources

About