Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation

Gan, Yulu; Ma, Xianzheng; Lou, Yihang; Yan, Binbin; Renrui, Zhang,; Shi, Ning; Luo, Lin

doi:10.48550/arxiv.2212.04145

Cited by 2 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Wang et al 2022a) serves as the first approach to tackle this task, using a combination of bi-average pseudo labels and stochastic weight reset. While (Wang et al 2022a;Song et al 2023) addresses the continual shifts at the model level, (Gan et al 2022a) leverages visual domain prompts to address the problem in the classification task at the input level for the first time. In this paper, we evaluate our approach on both TTA and CTTA with a specific focus on the dense prediction task.…”

Section: Related Workmentioning

confidence: 99%

“…The Thirty-Eighth AAAI Conference on Artificial Intelligence cent advances of prompting in NLP (Li and Liang 2021;Liu et al 2023), VDP (Gan et al 2022a) first introduces a prompt-based method to tackle the classification TTA problem. It employs image-level prompts to enhance domain transfer efficiency and effectiveness.…”

Section: Introductionmentioning

confidence: 99%

“…

The visual prompts have provided an efficient manner in addressing visual cross-domain problems. In previous works, ( Gan et al 2022a) first introduces domain prompts to tackle the classification Test-Time Adaptation (TTA) problem by placing image-level prompts on the input and fine-tuning prompts for each target domain. However, since the imagelevel prompts mask out continuous spatial details in the prompt-allocated region, it will suffer from inaccurate contextual information and limited domain knowledge extraction, particularly when dealing with dense prediction TTA problems.

…”

mentioning

confidence: 99%

See 2 more Smart Citations

Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction

Yang,

Wu,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

The visual prompts have provided an efficient manner in addressing visual cross-domain problems. Previous works introduce domain prompts to tackle the classification Test-Time Adaptation (TTA) problem by placing image-level prompts on the input and fine-tuning prompts for each target domain. However, since the image-level prompts mask out continuous spatial details in the prompt-allocated region, it will suffer from inaccurate contextual information and limited domain knowledge extraction, particularly when dealing with dense prediction TTA problems. To overcome these challenges, we propose a novel Sparse Visual Domain Prompts (SVDP) approach, which applies minimal trainable parameters (e.g., 0.1%) to pixels across the entire image and reserves more spatial information of the input. To better apply SVDP in extracting domain-specific knowledge, we introduce the Domain Prompt Placement (DPP) method to adaptively allocates trainable parameters of SVDP on the pixels with large distribution shifts. Furthermore, recognizing that each target domain sample exhibits a unique domain shift, we design Domain Prompt Updating (DPU) strategy to optimize prompt parameters differently for each sample, facilitating efficient adaptation to the target domain. Extensive experiments were conducted on widely-used TTA and continual TTA benchmarks, and our proposed method achieves state-of-the-art performance in both semantic segmentation and depth estimation tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

See 1 more Smart Citation

Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction

Yang,

Wu,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Specifically, CoTTA proposes to use the moving teacher model and augmentation-average predictions for noise suppression and the model stochastic restoration to avoid catastrophic forgetting. Following the scheme of CoTTA, some recent works [11,12,32] have addressed CTTA from different perspectives. Specifically, [12] leverages the temporal correlations of streamed input by reservoir sampling and instance-aware batch normalization.…”

Section: Related Workmentioning

confidence: 99%

“…Specifically, [12] leverages the temporal correlations of streamed input by reservoir sampling and instance-aware batch normalization. [11] proposes domain-specific prompts and domainagnostic prompts to preserve domain-specific and domainshared knowledge, respectively. EATA [32] performs adaptation on non-redundant samples for an efficient update.…”

Section: Related Workmentioning

confidence: 99%

Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation

Cao¹,

Xu²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Continual Test-Time Adaptation (CTTA) generalizes conventional Test-Time Adaptation (TTA) by assuming that the target domain is dynamic over time rather than stationary. In this paper, we explore Multi-Modal Continual Test-Time Adaptation (MM-CTTA) as a new extension of CTTA for 3D semantic segmentation. The key to MM-CTTA is to adaptively attend to the reliable modality while avoiding catastrophic forgetting during continual domain shifts, which is out of the capability of previous TTA or CTTA methods. To fulfill this gap, we propose an MM-CTTA method called Continual Cross-Modal Adaptive Clustering (CoMAC) that addresses this task from two perspectives. On one hand, we propose an adaptive dual-stage mechanism to generate reliable cross-modal predictions by attending to the reliable modality based on the class-wise featurecentroid distance in the latent space. On the other hand, to perform test-time adaptation without catastrophic forgetting, we design class-wise momentum queues that capture confident target features for adaptation while stochastically restoring pseudo-source features to revisit source knowledge. We further introduce two new benchmarks to facilitate the exploration of MM-CTTA in the future. Our experimental results show that our method achieves state-of-the-art performance on both benchmarks.

show abstract

Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation

Cited by 2 publications

References 0 publications

Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction

Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction

Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation

Contact Info

Product

Resources

About