Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer

Wu, Zhiyuan; Jiang, Yu; Zhao, Minghao; Cui, Chupeng; Zhan, Yang; Xue, Xinhui; Qi, Hong

doi:10.48550/arxiv.2104.14696

Cited by 1 publication

(1 citation statement)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our framework also intends to optimize the memory storage of each new domain, where we maximally reuse the parameters of the base generative model and only save the specific style modulation parameters (≈ 10MB) instead of the whole generative model (≈ 156MB). With recent advances in model compression and pruning [42], we believe the memory consumption could be further reduced in future.…”

Section: Discussionmentioning

confidence: 99%

Domain-incremental Cardiac Image Segmentation with Style-oriented Replay and Domain-sensitive Feature Whitening

Li,

Yu,

Heng

2022

Preprint

View full text Add to dashboard Cite

Contemporary methods have shown promising results on cardiac image segmentation, but merely in static learning, i.e., optimizing the network once for all, ignoring potential needs for model updating. In real-world scenarios, new data continues to be gathered from multiple institutions over time and new demands keep growing to pursue more satisfying performance. The desired model should incrementally learn from each incoming dataset and progressively update with improved functionality as time goes by. As the datasets sequentially delivered from multiple sites are normally heterogenous with domain discrepancy, each updated model should not catastrophically forget previously learned domains while well generalizing to currently arrived domains or even unseen domains. In medical scenarios, this is particularly challenging as accessing or storing past data is commonly not allowed due to data privacy. To this end, we propose a novel domainincremental learning framework to recover past domain inputs first and then regularly replay them during model optimization. Particularly, we first present a style-oriented replay module to enable structure-realistic and memoryefficient reproduction of past data, and then incorporate the replayed past data to jointly optimize the model with current data to alleviate catastrophic forgetting. During optimization, we additionally perform domain-sensitive feature whitening to suppress model's dependency on features that are sensitive to domain changes (e.g., domaindistinctive style features) to assist domain-invariant feature exploration and gradually improve the generalization performance of the network. We have extensively evaluated our approach with the M&Ms Dataset in single-domain and compound-domain incremental learning settings. Our approach outperforms other comparison methods with less forgetting on past domains and better generalization on current domains and unseen domains.

show abstract

Section: Discussionmentioning

confidence: 99%