An analysis of the exponentiated gradient descent algorithm

Hill, Sam; Williamson, Robert C.

doi:10.1109/isspa.1999.818191

Cited by 5 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The key challenge of CIL is that discarding old data is especially catastrophic for deep-learning models, as they are in the data-driven and end-to-end representation learning nature. The network parameters tuned by the old data will be overridden by the SGD optimizer using new data [11,33], causing a drastic performance dropcatastrophic forgetting [36,10,28] -on the old classes.…”

Section: Introductionmentioning

confidence: 99%

Distilling Causal Effect of Data in Class-Incremental Learning

Hu¹,

Tang²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation. We first 1) place CIL into the framework, 2) answer why the forgetting happens: the causal effect of the old data is lost in new training, and then 3) explain how the existing techniques mitigate it: they bring the causal effect back. Based on the framework, we find that although the feature/label distillation is storageefficient, its causal effect is not coherent with the end-toend feature learning merit, which is however preserved by data replay. To this end, we propose to distill the Colliding Effect between the old and the new data, which is fundamentally equivalent to the causal effect of data replay, but without any cost of replay storage. Thanks to the causal effect analysis, we can further capture the Incremental Momentum Effect of the data stream, removing which can help to retain the old effect overwhelmed by the new data effect, and thus alleviate the forgetting of the old class in testing. Extensive experiments on three CIL benchmarks: CIFAR-100, ImageNet-Sub&Full, show that the proposed causal effect distillation can improve various state-of-the-art CIL methods by a large margin (0.72%-9.06%). 1 1 There are also other settings like task-incremental [14,26].

show abstract

Section: Introductionmentioning

confidence: 99%

Distilling Causal Effect of Data in Class-Incremental Learning

Hu¹,

Tang²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 99%

Towards robust visual recognition: learning from imperfect data

Hu¹

View full text Add to dashboard Cite

of the most intelligent and self-disciplined people I have ever seen. It seems like yesterday when I think back to when he interviewed me for a Ph.D. position. I feel very fortunate to have worked with him during these years. He is not only a good mentor but also a good supporter. He always had a lot of ideas and unique insights, and he gave me the freedom to work on my interests at my own pace. He is always full of hope and enthusiasm and teaches me not to be timid. I would also like to thank my other paper collaborators: Yi Jiang, Kaihua Tang, Yulei Niu, and Jingyuan Chen. They accompanied me throughout most of my doctoral career, and they provided me with warm help at work and in life at every important moment.I also thank my co-supervisor, Prof. Chunyan Miao, who gives me much help in pursuing my Ph.D.

show abstract