Cross-domain Ensemble Distillation for Domain Generalization

Lee, Kyungmoon; Kim, Sungyeon; Kwak, Suha

doi:10.1007/978-3-031-19806-9_1

Cited by 19 publications

(3 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meta-learning optimizes a general domain-agnostic model, which turns into a domain-specific version with a few of the domain-specific samples for the test adaptation (Shu et al 2021;Chen et al 2023). Ensemble approaches integrated submodels with regards to diverse training domains to generalize unseen distributions (Lee, Kim, and Kwak 2022;Chu et al 2022).…”

Section: Related Workmentioning

confidence: 99%

Diagnosing and Rectifying Fake OOD Invariance: A Restructured Causal Approach

Chen,

Zheng,

Lai

et al. 2024

AAAI

View full text Add to dashboard Cite

Invariant representation learning (IRL) encourages the prediction from invariant causal features to labels deconfounded from the environments, advancing the technical roadmap of out-of-distribution (OOD) generalization. Despite spotlights around, recent theoretical result verified that some causal features recovered by IRLs merely pretend domain-invariantly in the training environments but fail in unseen domains. The fake invariance severely endangers OOD generalization since the trustful objective can not be diagnosed and existing causal remedies are invalid to rectify. In this paper, we review a IRL family (InvRat) under the Partially and Fully Informative Invariant Feature Structural Causal Models (PIIF SCM /FIIF SCM) respectively, to certify their weaknesses in representing fake invariant features, then, unify their causal diagrams to propose ReStructured SCM (RS-SCM). RS-SCM can ideally rebuild the spurious and the fake invariant features simultaneously. Given this, we further develop an approach based on conditional mutual information with respect to RS-SCM, then rigorously rectify the spurious and fake invariant effects. It can be easily implemented by a small feature selection subnet introduced in the IRL family, which is alternatively optimized to achieve our goal. Experiments verified the superiority of our approach to fight against the fake invariant issue across a variety of OOD generalization benchmarks.

show abstract

Section: Related Workmentioning

confidence: 99%

Diagnosing and Rectifying Fake OOD Invariance: A Restructured Causal Approach

Chen,

Zheng,

Lai

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Given this interesting setting and the promises of domain generalization in studying machine learning robustness, the community has developed a torrent of methods. Most of the existing methods fall into two categories: one is to build explicit regularization that pushes a model to learn representations that are invariant to the "style" across these domains [54][55][56][57][58][59][60][61][62][63][64][65][66][67][68][69][70][71][72]; the other one is to perform data augmentation that can introduce more diverse data to enrich the data of certain "semantic" information with the "style" from other domains [73][74][75][76][77][78][79][80], and also aims to train a model that is invariant to these "styles". More recently, there has been a line of approaches that aims to distill knowledge from pre-trained models into a smaller model to improve generalization performance [81][82][83][84][85].…”

Section: Domain Generalizationmentioning

confidence: 99%

Toward Out-of-Domain Binding Affinity Prediction

Zhao,

Fang,

Wang

2023

Preprint

View full text Add to dashboard Cite

Binding affinity prediction is pivotal in drug design, offering insights into the interactions between ligands and protein targets and thereby significantly influencing the drug development pipeline. Its potential to expedite the identification of drug candidates has led to extensive research focused on developing machine learning algorithms for predicting binding affinity. However, most developments have concentrated on independently and identically distributed (i.i.d) data. In real-world scenarios, prediction models may encounter novel chemical substructures, protein families absent from the training set, variations in experimental conditions, and evolving drug resistance mechanisms. These factors can lead to a significant degradation in performance, causing models to suggest suboptimal compounds or overlook promising candidates—challenges commonly referred to as Out-of-Domain (OOD) in the machine learning community. To address the OOD challenges in binding affinity algorithm development, several benchmarks have been introduced. However, we observe that many lack a convenient codebase framework for swift algorithm evaluation.In this paper, building upon the DrugOOD dataset, we introduce a comprehensive benchmarking framework to assess the resilience and adaptability of OOD algorithms in binding affinity prediction. Our framework offers a streamlined approach for evaluating algorithmic performance in OOD scenarios. Furthermore, we propose a method that surpasses existing state-of-the-art approaches in our benchmark tests. We anticipate that our contributions will spur further research addressing OOD challenges and enhance the reliability and robustness of binding affinity predictions in drug design. Code available at: https://github.com/zehanzz/BioFrontierOOD.git

show abstract

“…Compared to domain adaptation (DA), it is more practical but also more challenging since the target images do not participate in training. The mainstream approaches generally can be divided into the following categories: domain-invariant representation learning [20][21][22], data augmentation [23][24][25], meta-learning [26][27][28], and ensemble learning [29][30][31]. For domaininvariant representation learning, adversarial learning [32] and minimize maximum mean discrepancy (MMD) [33] are the more popular strategies for the extraction of invariant information.…”

Section: Domain Generalizationmentioning

confidence: 99%

A Dual-Attention Deep Discriminative Domain Generalization Model for Hyperspectral Image Classification

Zhao,

Wang,

Wang

et al. 2023

Remote Sensing

View full text Add to dashboard Cite

Recently, hyperspectral image classification has made great progress with the development of convolutional neural networks. However, due to the challenges of distribution shifts and data redundancies, the classification accuracy is low. Some existing domain adaptation methods try to mitigate the distribution shifts by training source samples and some labeled target samples. However, in practice, labeled target domain samples are difficult or even impossible to obtain. To solve the above challenges, we propose a novel dual-attention deep discriminative domain generalization framework (DAD3GM) for cross-scene hyperspectral image classification without training the labeled target samples. In DAD3GM, we mainly design two blocks: dual-attention feature learning (DAFL) and deep discriminative feature learning (DDFL). DAFL is designed to extract spatial features by multi-scale self-attention and extract spectral features by multi-head external attention. DDFL is further designed to extract deep discriminative features by contrastive regularization and class discrimination regularization. The combination of DAFL and DDFL can effectively reduce the computational time and improve the generalization performance of DAD3GM. The proposed model achieves 84.25%, 83.53%, and 80.63% overall accuracy on the public Houston, Pavia, and GID benchmarks, respectively. Compared with some classical and state-of-the-art methods, the proposed model achieves optimal results, which reveals its effectiveness and feasibility.

show abstract

Cross-domain Ensemble Distillation for Domain Generalization

Cited by 19 publications

References 60 publications

Diagnosing and Rectifying Fake OOD Invariance: A Restructured Causal Approach

Diagnosing and Rectifying Fake OOD Invariance: A Restructured Causal Approach

Toward Out-of-Domain Binding Affinity Prediction

A Dual-Attention Deep Discriminative Domain Generalization Model for Hyperspectral Image Classification

Contact Info

Product

Resources

About