Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers

Ye, Jingwen; Wang, Xinchao; Ji, Yixin; Ou, Kairi; Song, Mingli

doi:10.24963/ijcai.2019/573

Cited by 25 publications

(21 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to handle the multi-task problem in one single network, the work of [33] proposes an effective method to train the student network on multiple scene understanding tasks, which leads to better performance than the teachers. To make it further, Ye et al [35] apply a two-step filter strategy to customize the arbitrary task set on TargetNet.…”

Section: Related Workmentioning

confidence: 99%

“…Motivated by the work of [35], we use the block-wise training strategy to transfer as much knowledge as possible from the generator G into the dual-generator T . That is, we divide dual-generator into B blocks as {T Then the task-level filtering g m is applied to meet the task customizable demand and embedded into the m-th branch block-wise adversarial loss:…”

Section: Dual-generator Trainingmentioning

confidence: 99%

“…Besides, the seminal work of knowledge distillation (KD) [13] learns a compact student model by the aid of the soft labels obtained from the teachers. Other than these works that learn from one single network, knowledge amalgamation (KA) [35] is proposed to effectively reuse multiple pre-trained networks, and make the learned multiknowledge settle well in one single network. Besides, the process of KA doesn't require any annotations, where only the intermediate features are treated as knowledge to guide training.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

Wang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

Recent advances in deep learning have provided procedures for learning one network to amalgamate multiple streams of knowledge from the pre-trained Convolutional Neural Network (CNN) models, thus reduce the annotation cost. However, almost all existing methods demand massive training data, which may be unavailable due to privacy or transmission issues. In this paper, we propose a datafree knowledge amalgamate strategy to craft a well-behaved multi-task student network from multiple single/multi-task teachers. The main idea is to construct the group-stack generative adversarial networks (GANs) which have two dual generators. First one generator is trained to collect the knowledge by reconstructing the images approximating the original dataset utilized for pre-training the teachers. Then a dual generator is trained by taking the output from the former generator as input. Finally we treat the dual part generator as the target network and regroup it. As demonstrated on several benchmarks of multi-label classification, the proposed method without any training data achieves the surprisingly competitive results, even compared with some full-supervised methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Dual-generator Trainingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

Wang

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

show abstract

“…It adopts an auto- encoder architecture to amalgamate features from multiple single-task teachers. Several knowledge amalgamation methods are also proposed to handle the above task [33,23,34]. The proposed approach here, on the other hand, handles teachers working on both single or multiple tasks, and follows a dual-stage strategy tailored for customizing the student network that also gives rise to component nets as byproducts.…”

Section: Related Workmentioning

confidence: 99%

Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation

Shen

Xue

Wang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

A massive number of well-trained deep networks have been released by developers online. These networks may focus on different tasks and in many cases are optimized for different datasets. In this paper, we study how to exploit such heterogeneous pre-trained networks, known as teachers, so as to train a customized student network that tackles a set of selective tasks defined by the user. We assume no human annotations are available, and each teacher may be either single-or multi-task. To this end, we introduce a dualstep strategy that first extracts the task-specific knowledge from the heterogeneous teachers sharing the same sub-task, and then amalgamates the extracted knowledge to build the student network. To facilitate the training, we employ a selective learning scheme where, for each unlabelled sample, the student learns adaptively from only the teacher with the least prediction ambiguity. We evaluate the proposed approach on several datasets and experimental results demonstrate that the student, learned by such adaptive knowledge amalgamation, achieves performances even better than those of the teachers.

show abstract

“…Various techniques have also been proposed to improve the generalisability and the performance of the student models including using noisy data (Li et al, 2017; Sarfraz et al, 2019), adaptive regularisation of distillation parameters (Ding et al, 2019) and adversarial perturbation of data for training (Xie et al, 2020). Multi-task learning methods have also been shown to provide a good regularisation, reducing the risk of over-fitting (Liu et al, 2019a; Ye et al, 2019). The auxiliary task could be a related task (e.g.…”

Section: Introductionmentioning

confidence: 99%

Automated Detection of Cerebral Microbleeds on MR images using Knowledge Distillation Framework

Sundaresan

Arthofer

Zamboni

et al. 2021

Preprint

View full text Add to dashboard Cite

Cerebral microbleeds (CMBs) are associated with white matter damage, various neu-rodegenerative and cerebrovascular diseases. CMBs occur as small, circular hypointense lesions on T2*-weighted gradient recalled echo (GRE) and susceptibility weighted imaging (SWI) images, and hyperintense on quantitative susceptibility mapping (QSM) images due to their paramagnetic nature. Accurate detection of CMBs would help to determine the CMB lesion count and distribution, which would be further useful to understand the clinical impact of CMBs and to obtain quantitative imaging biomarkers. In this work, we propose a fully automated, deep learning-based, 2-step algorithm, using structural and anatomical properties of CMBs from any single input image modality (e.g. GRE / SWI / QSM) for their accurate detection. Our method consists of an initial candidate detection step, that detects CMBs with high sensitivity and a candidate discrimination step using a knowledge distillation framework to classify CMB and non-CMB instances, followed by a morphological clean-up step. We used 4 datasets consisting of different modalities specified above, acquired using various protocols and with a variety of pathological and demographic characteristics. On cross-validation within datasets, our method achieved a cluster-wise true positive rate (TPR) over 90% with an average of less than 2 false positives per subject. Our method is flexible in terms of the input modality and provides comparable cluster-wise TPR and a better cluster-wise precision compared to existing state-of-the-art methods. When evaluated across different datasets, our method showed good generalisability with a cluster-wise TPR greater than 80% with different modalities.

show abstract

Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers

Cited by 25 publications

References 6 publications

Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation

Automated Detection of Cerebral Microbleeds on MR images using Knowledge Distillation Framework

Contact Info

Product

Resources

About