Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models

Özdenizci, Ozan; Legenstein, Robert

doi:10.1109/tpami.2023.3238179

Cited by 132 publications

(44 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We can solve the cross-modality data translation problem by adapting the target data F generation task into the framework of score-matching and then using a perturbed source domain G to guide (conditioning) the iterative diffusion process [41,46,7,47,73,32,6]. Ideally, we want the generated data F to follow the semantic meaning of the guided data G and share the features as the ground truth data point F in the target domain, balancing realism and faithfulness.…”

Section: Diffusion For Cross-modality Data Translationmentioning

confidence: 99%

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Wang¹,

Yingyu²,

Sermesant³

et al. 2023

Preprint

View full text Add to dashboard Cite

Cross-modality data translation has attracted great interest in image computing. Deep generative models (e.g., GANs) show performance improvement in tackling those problems. Nevertheless, as a fundamental challenge in image translation, the problem of Zero-shot-Learning Cross-Modality Data Translation with fidelity remains unanswered. This paper proposes a new unsupervised zero-shot-learning method named Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion), which learns to translate the unseen source data to the target domain. The MIDiffusion leverages a score-matching-based generative model, which learns the prior knowledge in the target domain. We propose a differentiable local-wise-MI-Layer (LM I) for conditioning the iterative denoising sampling. The LM I captures the identical cross-modality features in the statistical domain for the diffusion guidance; thus, our method does not require retraining when the source domain is changed, as it does not rely on any direct mapping between the source and target domains. This advantage is critical for applying cross-modality data translation methods in practice, as a reasonable amount of source domain dataset is not always available for supervised training. We empirically show the advanced performance of MIDiffusion in comparison with an influential group of generative models, including adversarial-based and other scorematching-based models.

show abstract

Section: Diffusion For Cross-modality Data Translationmentioning

confidence: 99%

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Wang¹,

Yingyu²,

Sermesant³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Existing DDPM-based image enhancement methods usually directly employ the original degraded image as condition [20], [54], [55]. However, such a strategy may not suit dehazing task since the dense-haze images suffer from huge haze-induced information loss in content and color and the distribution of the original hazy input severely deviates from that of the clear image, as shown in Fig.…”

Section: Backbone Of Fsdgnmentioning

confidence: 99%

“…In contrast, diffusion models, as a class of likelihoodbased generative models, possess the desirable properties such as distribution coverage, a stationary training objective, and easy scalability [16]- [19]. With this line, conditional DDPM [20]- [22], [53]- [55] are developed for image enhancement in low-level vision, such as image super-resolution [20], image inpainting [54], and image deblurring [21]. Although DDPMbased methods have been developed for some low-level vision tasks, there is no precedent for usage in image dehazing.…”

Section: Introductionmentioning

confidence: 99%

Frequency and Spatial Dual Guidance for Image Dehazing

Zheng

Zhou

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Image dehazing is quite challenging in dense-haze scenarios, where quite less original information remains in the hazy image. Though previous methods have made marvelous progress, they still suffer from information loss in content and color in dense-haze scenarios. The recently emerged Denoising Diffusion Probabilistic Model (DDPM) exhibits strong generation ability, showing potential for solving this problem. However, DDPM fails to consider the physics property of dehazing task, limiting its information completion capacity. In this work, we propose DehazeDDPM: A DDPM-based and physics-aware image dehazing framework that applies to complex hazy scenarios. Specifically, DehazeDDPM works in two stages. The former stage physically models the dehazing task with the Atmospheric Scattering Model (ASM), pulling the distribution closer to the clear data and endowing DehazeDDPM with fog-aware ability. The latter stage exploits the strong generation ability of DDPM to compensate for the haze-induced huge information loss, by working in conjunction with the physical modelling. Extensive experiments demonstrate that our method attains state-of-theart performance on both synthetic and real-world hazy datasets.

show abstract

“…Lately, DDPMs have shown to be a promising approach for the task of UAD in brain MRI as they have scalable and stable training properties while generating sharp images of high quality (Wolleb et al, 2022;Wyatt et al, 2022;Sanchez et al, 2022;Pinaya et al, 2022a). While these approaches aim to estimate the entire brain anatomy at once, patch-based DDPMs have been proposed for image restoration ( Özdenizci and Legenstein, 2023) and image inpainting (Lugmayr et al, 2022) in the domain of generic images. Patch-based DDPMs are a promising approach also for brain MRI reconstruction, as global context information about individual brain structure and appearance could be incorporated while estimating individual patches.…”

Section: Recent Workmentioning

confidence: 99%

“…Patch-based DDPMs are a promising approach also for brain MRI reconstruction, as global context information about individual brain structure and appearance could be incorporated while estimating individual patches. However, current patch-based approaches either neglect the surrounding context of each patch ( Özdenizci and Legenstein, 2023) or reconstruct patches from a fully noised image, which also impacts the surrounding context (Lugmayr et al, 2022). Thus, it is of interest to develop patch-based DDPMs that consider both the individual patch and its unperturbed surrounding context for the task of UAD in brain MRI.…”

Section: Recent Workmentioning

confidence: 99%

Unsupervised Anomaly Detection in 3D Brain MRI Using Deep Learning with Impured Training Data

Behrendt¹,

Bengs²,

Frederik³

et al. 2022

2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)

View full text Add to dashboard Cite

The detection of lesions in magnetic resonance imaging (MRI)-scans of human brains remains challenging, timeconsuming and error-prone. Recently, unsupervised anomaly detection (UAD) methods have shown promising results for this task. These methods rely on training data sets that solely contain healthy samples. Compared to supervised approaches, this significantly reduces the need for an extensive amount of labeled training data. However, data labelling remains error-prone. We study how unhealthy samples within the training data affect anomaly detection performance for brain MRI-scans. For our evaluations, we consider three publicly available data sets and use autoencoders (AE) as a well-established baseline method for UAD. We systematically evaluate the effect of impured training data by injecting different quantities of unhealthy samples to our training set of healthy samples from T1-weighted MRI-scans. We evaluate a method to identify falsely labeled samples directly during training based on the reconstruction error of the AE. Our results show that training with impured data decreases the UAD performance notably even with few falsely labeled samples. By performing outlier removal directly during training based on the reconstruction-loss, we demonstrate that falsely labeled data can be detected and removed to mitigate the effect of falsely labeled data. Overall, we highlight the importance of clean data sets for UAD in brain MRI and demonstrate an approach for detecting falsely labeled data directly during training.

show abstract

Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models

Cited by 132 publications

References 58 publications

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Frequency and Spatial Dual Guidance for Image Dehazing

Unsupervised Anomaly Detection in 3D Brain MRI Using Deep Learning with Impured Training Data

Contact Info

Product

Resources

About