Balanced softmax cross-entropy for incremental learning with and without memory

Jodelet, Quentin; Liu, Xin; Murata, Tsuyoshi

doi:10.1016/j.cviu.2022.103582

Cited by 11 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Utilizing pretrained models for parameter-efficient finetuning is gaining attention due to their strong feature extraction capabilities [166], [167], [59], [168]. Generating pseudodata of previous tasks using generative models like diffusion model [169], [170] still provides a promising direction. Beyond conventional forgetting-related research, combining continual learning with other learning paradigms represents an important trend.…”

Section: Algorithmmentioning

confidence: 99%

Continual Learning with Knowledge Distillation: A Survey

Li,

Su,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

The foremost challenge in continual learning is devising strategies to alleviate catastrophic forgetting, thereby preserving a model’s memory of prior knowledge while learning new tasks. Knowledge distillation, a form of data regularization, is garnering increasing attention in the field of continual learning for its ability to constrain a model’s discriminative power for previous tasks by emulating the outputs of old task models while learning new tasks, thus mitigating forgetting. This paper offers a comprehensive survey of continual learning methods employing knowledge distillation within the realm of image classification. We inductively categorize these methods according to the source of knowledge utilized and provide a detailed analysis of the distillation solutions they employ. Furthermore, given the characteristic inability of continual learning to access historical data, we introduce a novel taxonomy for continual learning approaches from the perspective of auxiliary data usage. In addition, we have conducted extensive experiments on CIFAR-100, TinyImageNet, and ImageNet-100 across nine knowledge distillation-integrated continual learning methods, deeply analyzing the role of knowledge distillation in different continual learning scenarios to alleviate model forgetting. Our substantial experimental evidence demonstrates that knowledge distillation can indeed reduce forgetting across most scenarios.

show abstract

Section: Algorithmmentioning

confidence: 99%

Continual Learning with Knowledge Distillation: A Survey

Li,

Su,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Adding more instances of the less dominant class to training data might potentially solve the problem. Therefore, we propose to use the Balanced Cross-Entropy loss (BCE) function [ 34 ] as in Equation (1):

where

is the class SoftMax probability and

is the ground truth of the corresponding prediction.

and

presents the total of pixels in the image.…”

Section: Lightweight Semantic Segmentation Fcn-mobilenetv2mentioning

confidence: 99%

Section: Model Trainingmentioning

confidence: 99%

See 1 more Smart Citation

IRDC-Net: Lightweight Semantic Segmentation Network Based on Monocular Camera for Mobile Robot Navigation

Dang

Tran

Tan

2023

Sensors

View full text Add to dashboard Cite

Computer vision plays a significant role in mobile robot navigation due to the wealth of information extracted from digital images. Mobile robots localize and move to the intended destination based on the captured images. Due to the complexity of the environment, obstacle avoidance still requires a complex sensor system with a high computational efficiency requirement. This study offers a real-time solution to the problem of extracting corridor scenes from a single image using a lightweight semantic segmentation model integrating with the quantization technique to reduce the numerous training parameters and computational costs. The proposed model consists of an FCN as the decoder and MobilenetV2 as the decoder (with multi-scale fusion). This combination allows us to significantly minimize computation time while achieving high precision. Moreover, in this study, we also propose to use the Balance Cross-Entropy loss function to handle diverse datasets, especially those with class imbalances and to integrate a number of techniques, for example, the Adam optimizer and Gaussian filters, to enhance segmentation performance. The results demonstrate that our model can outperform baselines across different datasets. Moreover, when being applied to practical experiments with a real mobile robot, the proposed model’s performance is still consistent, supporting the optimal path planning, allowing the mobile robot to efficiently and effectively avoid the obstacles.

show abstract