Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation

Nguyen-Meidine, Le Thanh; Granger, Éric; Kiran, Madhu; Dolz, José; Blais-Morin, Louis-Antoine

doi:10.1109/ijcnn48605.2020.9206989

Cited by 21 publications

(7 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While [30] achieves good performance for MTDA, we argue that it does not take full advantage of the current UDA techniques and both these techniques require a lot of resources. In addition, since it only uses the source domain for distillation which has been shown in [21] to have limited knowledge transfer and can reduce accuracy. Also, This distillation with source only guarantees consistency w.r.t to source domain and does not guarantee that features will remain domaininvariant with previous target domains.…”

Section: Multi-target Domain Adaptationmentioning

confidence: 99%

Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer

Nguyen-Meidine¹,

Kiran²,

Pedersoli³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Techniques for multi-target domain adaptation (MTDA) seek to adapt a recognition model such that it can generalize well across multiple target domains. While several successful techniques have been proposed for unsupervised single-target domain adaptation (STDA) in object detection, adapting a model to multiple target domains using unlabeled image data remains a challenging and largely unexplored problem. Key challenges include the lack of bounding box annotations for target data, knowledge corruption, and the growing resource requirements needed to train accurate deep detection models. The later requirements are augmented by the need to retraining a model with previous-learned target data when adapting to each new target domain. Currently, the only MTDA technique in literature for object detection relies on distillation with a duplicated model to avoid knowledge corruption but does not leverage the source-target feature alignment after UDA. To address these challenges, we propose a new Incremental MTDA technique for object detection that can adapt a detector to multiple target domains, one at a time, without having to retain data of previously-learned target domains. Instead of distillation, our technique efficiently transfers source images to a joint target domains' space, on the fly, thereby preserving knowledge during incremental MTDA. Using adversarial training, our Domain Transfer Module (DTM) is optimized to trick the domain classifiers into classifying source images as though transferred into the target domain, thus allowing the DTM to generate samples close to a joint distribution of target domains.Our proposed technique is validated on different MTDA detection benchmarks, and results show it improving accuracy across multiple domains, despite the considerable reduction in complexity. Our code is available at: link 1

show abstract

Section: Multi-target Domain Adaptationmentioning

confidence: 99%

Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer

Nguyen-Meidine¹,

Kiran²,

Pedersoli³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Previous Ensemble-Distillation Method: In this section, we explain how the concepts of the previous ensembledistillation works [42][43][44][45][46][47][48][49] can be borrowed to perform semantic segmentation based UDA ensemble-distillation tasks. Typically, these works view T as a set of probabilistic models, and complete the ensemble-distillation process through minimizing the negative log-likelihood loss L KL between the expected outputs from the ensemble and the student model, as depicted in Fig.…”

Section: Preliminarymentioning

confidence: 99%

“…To address such a problem, the concept of ensembledistillation [42][43][44][45][46][47][48][49][50] can be leveraged since its focus is on designing an effective distillation process instead of a costly end-to-end ensemble learning framework. Typically, these ensemble distillation frameworks view the members in an ensemble as probabilistic models, and transfer the knowledge using expected certainty outputs.…”

Section: Introductionmentioning

confidence: 99%

Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaptation

Chao,

Cheng,

Lee

2021

Preprint

View full text Add to dashboard Cite

Recent researches on unsupervised domain adaptation (UDA) have demonstrated that end-to-end ensemble learning frameworks serve as a compelling option for UDA tasks. Nevertheless, these end-to-end ensemble learning methods often lack flexibility as any modification to the ensemble requires retraining of their frameworks. To address this problem, we propose a flexible ensemble-distillation framework for performing semantic segmentation based UDA, allowing any arbitrary composition of the members in the ensemble while still maintaining its superior performance. To achieve such flexibility, our framework is designed to be robust against the output inconsistency and the performance variation of the members within the ensemble. To examine the effectiveness and the robustness of our method, we perform an extensive set of experiments on both GTA5→Cityscapes and SYNTHIA→Cityscapes benchmarks to quantitatively inspect the improvements achievable by our method. We further provide detailed analyses to validate that our design choices are practical and beneficial. The experimental evidence validates that the proposed method indeed offer superior performance, robustness and flexibility in semantic segmentation based UDA tasks against contemporary baseline methods.

show abstract

“…Several works [13] have been proposed in the classification paradigm to learn the representations of out-of-distribution data. Particularly [14,15] utilize a knowledge distillation strategy in image classification to learn out-of-distribution data. However, to the best of our knowledge only [16] has proposed a domain adaptation strategy in a regression setting to improve keypoint detection.…”

Section: Introductionmentioning

confidence: 99%

Towards Accurate Cross-Domain In-Bed Human Pose Estimation

Afham¹,

Haputhanthri²,

Pradeepkumar³

et al. 2021

Preprint

View full text Add to dashboard Cite

Human behavioral monitoring during sleep is essential for various medical applications. Majority of the contactless human pose estimation algorithms are based on RGB modality, causing ineffectiveness in in-bed pose estimation due to occlusions by blankets and varying illumination conditions. Long-wavelength infrared (LWIR) modality based pose estimation algorithms overcome the aforementioned challenges; however, ground truth pose generations by a human annotator under such conditions are not feasible. A feasible solution to address this issue is to transfer the knowledge learned from images with pose labels and no occlusions, and adapt it towards real world conditions (occlusions due to blankets). In this paper, we propose a novel learning strategy comprises of two-fold data augmentation to reduce the cross-domain discrepancy and knowledge distillation to learn the distribution of unlabeled images in real world conditions. Our experiments and analysis show the effectiveness of our approach over multiple standard human pose estimation baselines. Our code is available at: https://github.com/MohamedAfham/CD_HPE.

show abstract

Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation

Cited by 21 publications

References 13 publications

Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer

Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer

Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaptation

Towards Accurate Cross-Domain In-Bed Human Pose Estimation

Contact Info

Product

Resources

About