Test-Time Fast Adaptation for Dynamic Scene Deblurring via Meta-Auxiliary Learning

Chi, Zhixiang; Wang, Yang; Yu, Yuanhao; Tang, Jin

doi:10.1109/cvpr46437.2021.00902

Cited by 54 publications

(23 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, given a test sample, TTT will first train the model using this sample with self-supervisions and then use the updated model for final prediction. After that, the idea of TTT has been applied to many real-world applications, such as human pose estimation [12], dynamic scene deblurring [3], and longtailed learning [48]. Compared with TTT that is designed for improving the model performance on out-of-distribution (OOD) test samples, our method is more general as it 1) not only can be applied to existing OOD generalization methods to further boost the performance 2) but also improves the predictive performance of any pre-trained model on test samples that have the same distribution with training samples.…”

Section: Related Workmentioning

confidence: 99%

“…Threshold in Eqn. (3). We evaluate CLI with different , selected from {0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}, based on ResNet-18.…”

Section: Ablationsmentioning

confidence: 99%

“…Second, at the second loop of predicting hard test samples, how to improve the model's predictive ability to enable it to correct those erroneous predictions is unclear. Test time learning-based methods [3,11,32,39] exploit test samples to conduct self-supervised training (e.g., rotation prediction [7]) before making the final prediction. However, these methods are specially designed for overcoming the distribution shift between training and test data.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Boost Test-Time Performance with Closed-Loop Inference

Niu¹,

Wu²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Conventional deep models predict a test sample with a single forward propagation, which, however, may not be sufficient for predicting hard-classified samples. On the contrary, we human beings may need to carefully check the sample many times before making a final decision. During the recheck process, one may refine/adjust the prediction by referring to related samples. Motivated by this, we propose to predict those hard-classified test samples in a looped manner to boost the model performance. However, this idea may pose a critical challenge: how to construct looped inference, so that the original erroneous predictions on these hard test samples can be corrected with little additional effort. To address this, we propose a general Closed-Loop Inference (CLI) method. Specifically, we first devise a filtering criterion to identify those hard-classified test samples that need additional inference loops. For each hard sample, we construct an additional auxiliary learning task based on its original top-K predictions to calibrate the model, and then use the calibrated model to obtain the final prediction. Promising results on ImageNet (in-distribution test samples) and ImageNet-C (out-of-distribution test samples) demonstrate the effectiveness of CLI in improving the performance of any pre-trained model.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Threshold in Eqn. (3). We evaluate CLI with different , selected from {0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}, based on ResNet-18.…”

Section: Ablationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Boost Test-Time Performance with Closed-Loop Inference

Niu¹,

Wu²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Test-Time Adaptation (TTA) aims to enable quick adaptation of an existing model to new target data without having access to the source domain data the model was trained on. As an important challenge for dealing with dynamic domain shift in real-world, TTA is attracting more and more attention in several tasks [4,15,19,25,31]. Among them, Test-time Training (TTT) [25] updates model parameters in an online manner by applying a self-supervised proxy task on the test data.…”

Section: Related Workmentioning

confidence: 99%

MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation

Shin¹,

Tsai²,

Zhuang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that, directly applying existing methods usually results in performance instability at test time, because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra-modal Pseudolabel Generation (Intra-PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre-trained on source data but updated with target data at different paces. Second, Inter-modal Pseudo-label Refinement (Inter-PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self-learning signals in numerous multi-modal test-time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.neclabs.com/ ˜mas/MM-TTA

show abstract

“…Cho et al [8] presented a multi-input multi-output U-net (MIMO-UNet) which utilizes a single U-Net (i.e., encoder-decoder with short connections) but multiple input and output images to handle the coarse-to-fine image deblurring. Chi et al [7] utilized an encoder-decoder network to extract multiscale image features, and then integrated the auxiliary and meta learning to enhance the deblurring performance. Chen et al [4] also applied encoder-decoder architecture to implement multi-scale and multi-stage image restoration tasks by introducing a new normalization method.…”

Section: Related Work 21 Image Deblurringmentioning

confidence: 99%

Multi-scale frequency separation network for image deblurring

Zhang¹,

Li²,

Miao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Image deblurring aims to restore the detailed texture information or structures from the blurry images, which has become an indispensable step in many computer-vision tasks. Although various methods have been proposed to deal with the image deblurring problem, most of them treated the blurry image as a whole and neglected the characteristics of different image frequencies. In this paper, we present a new method called multi-scale frequency separation network (MSFS-Net) for image deblurring. MSFS-Net introduces the frequency separation module (FSM) into an encoder-decoder network architecture to capture the lowand high-frequency information of image at multiple scales. Then, a simple cycle-consistency strategy and a sophisticated contrastive learning module (CLM) are respectively designed to retain the low-frequency information and recover the high-frequency information during deblurring. At last, the features of different scales are fused by a crossscale feature fusion module (CSFFM). Extensive experiments on benchmark datasets show that the proposed network achieves state-of-the-art performance.

show abstract

Test-Time Fast Adaptation for Dynamic Scene Deblurring via Meta-Auxiliary Learning

Cited by 54 publications

References 33 publications

Boost Test-Time Performance with Closed-Loop Inference

Boost Test-Time Performance with Closed-Loop Inference

MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation

Multi-scale frequency separation network for image deblurring

Contact Info

Product

Resources

About