Attention Diversification for Domain Generalization

Abstract

Convolutional neural networks (CNNs) have demonstrated gratifying results at learning discriminative features. However, when applied to unseen domains, state-of-the-art models are usually prone to errors due to domain shift. After investigating this issue from the perspective of shortcut learning, we find the devils lie in the fact that models trained on different domains merely bias to different domain-specific features yet overlook diverse task-related features. Under this guidance, a novel Attention Diversification framework is proposed, in which Intra-Model and Inter-Model Attention Diversification Regularization are collaborated to reassign appropriate attention to diverse task-related features. Briefly, Intra-Model Attention Diversification Regularization is equipped on the high-level feature maps to achieve in-channel discrimination and cross-channel diversification via forcing different channels to pay their most salient attention to different spatial locations. Besides, Inter-Model Attention Diversification Regularization is proposed to further provide task-related attention diversification and domain-related attention suppression, which is a paradigm of "simulate, divide and assemble": simulate domain shift via exploiting multiple domain-specific models, divide attention maps into task-related and domain-related groups, and assemble them within each group respectively to execute regularization. Extensive experiments and analyses are conducted on various benchmarks to demonstrate that our method achieves state-of-the-art performance over other competing methods. Code is available at https: //github.com/hikvision-research/DomainGeneralization.

Topology-Driven Negative Sampling Enhances Generalizability in Protein-Protein Interaction Prediction

“…To develop a generalizable AI model, learning PPI from the protein structures instead of leveraging the topology that inflates the performance, we need to mitigate shortcut learning and make protein representation learning independent of the PPI training data. Furthermore, regularization has been proven to be an important tool for improving generalizability in computer vision [18]. Yet, regularization is based on the assumption that similarity between a pair of entities implies interaction, an assumption that is true in image processing and social sciences [19], yet similar molecules do not necessarily always interact in PPI and other biological systems [20].…”

A Dynamic Learning Method towards Realistic Compositional Zero-Shot Learning

“…CycleMAE (Yang et al 2023) uses a Masked Auto-Encoder (MAE) model to create a cycle image reconstruction task that results in more realistic and unique image pairs. Other typical domain generalization methods generally include model ensemble (Zhou et al 2021;Qu et al 2022), feature normalization (Zhu et al 2022;Meng et al 2022) or contrastive learning techniques (Yao et al 2022;Li et al 2023a).…”

Toward Out-of-Domain Binding Affinity Prediction

“…Given this interesting setting and the promises of domain generalization in studying machine learning robustness, the community has developed a torrent of methods. Most of the existing methods fall into two categories: one is to build explicit regularization that pushes a model to learn representations that are invariant to the "style" across these domains [54][55][56][57][58][59][60][61][62][63][64][65][66][67][68][69][70][71][72]; the other one is to perform data augmentation that can introduce more diverse data to enrich the data of certain "semantic" information with the "style" from other domains [73][74][75][76][77][78][79][80], and also aims to train a model that is invariant to these "styles". More recently, there has been a line of approaches that aims to distill knowledge from pre-trained models into a smaller model to improve generalization performance [81][82][83][84][85].…”