Posterior Sampling for Image Restoration using Explicit Patch Priors

Friedman, Roy; Weiss, Yair

doi:10.48550/arxiv.2104.09895

Cited by 3 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When is posterior sampling optimal? Many recent image restoration methods attempt to produce diverse high perceptual quality reconstructions by sampling from the posterior distribution p X|Y [7,18,10]. As discussed in [4], the posterior sampling estimator attains a perception index of 0 (namely W 2 (p X , p X ) = 0) and distortion 2D * .…”

Section: The Wasserstein and Gelbrich Distancesmentioning

confidence: 99%

“…3.1). Further simplification arises when γ, µ are centered non-singular Gaussian measures, in which case T γ→µ is the linear and symmetric transformation (7). Then, γ t is a Gaussian measure with covariance Σ γt = T t Σ γ T t , where T t [I + t(T γ→µ − I)].…”

Section: A Geometric Perspective On the Distortion-perception Tradeoffmentioning

confidence: 99%

“…This is done e.g. using priors over image patches [7], conditional generative models [18,20], or implicit priors induced by deep denoiser networks [10]. Theoretically, posterior sampling leads to perfect perceptual quality (the restored outputs are distributed like the prior).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Theory of the Distortion-Perception Tradeoff in Wasserstein Space

Freirich¹,

Michaeli²,

Meir³

2021

Preprint

View full text Add to dashboard Cite

The lower the distortion of an estimator, the more the distribution of its outputs generally deviates from the distribution of the signals it attempts to estimate. This phenomenon, known as the perception-distortion tradeoff, has captured significant attention in image restoration, where it implies that fidelity to ground truth images comes at the expense of perceptual quality (deviation from statistics of natural images). However, despite the increasing popularity of performing comparisons on the perception-distortion plane, there remains an important open question: what is the minimal distortion that can be achieved under a given perception constraint? In this paper, we derive a closed form expression for this distortion-perception (DP) function for the mean squared-error (MSE) distortion and the Wasserstein-2 perception index. We prove that the DP function is always quadratic, regardless of the underlying distribution. This stems from the fact that estimators on the DP curve form a geodesic in Wasserstein space. In the Gaussian setting, we further provide a closed form expression for such estimators. For general distributions, we show how these estimators can be constructed from the estimators at the two extremes of the tradeoff: The global MSE minimizer, and a minimizer of the MSE under a perfect perceptual quality constraint. The latter can be obtained as a stochastic transformation of the former.Preprint. Under review.

show abstract

Section: The Wasserstein and Gelbrich Distancesmentioning

confidence: 99%

Section: A Geometric Perspective On the Distortion-perception Tradeoffmentioning

confidence: 99%

See 1 more Smart Citation

A Theory of the Distortion-Perception Tradeoff in Wasserstein Space

Freirich¹,

Michaeli²,

Meir³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, [93] proposed the negative log likelihood of all patches of an image as a regularizer, where the underlying patch distribution was assumed to follow a Gaussian mixture model (GMM) which parameters were learned from few clean images. This method is still competitive to many approaches based on deep learning and several extensions were suggested recently [22,61,72]. However, even though GMMs can approximate any probability density function if the number of components is large enough, they suffer from limited flexibility in case of a fixed number of components, see [23] and the references therein.…”

Section: Introductionmentioning

confidence: 99%

PatchNR: learning from very few images by patch normalizing flow regularization

et al. 2023

View full text Add to dashboard Cite

Learning neural networks using only few available information is an important ongoing research topic with tremendous potential for applications. In this paper, we introduce a powerful regularizer for the variational modeling of inverse problems in imaging. Our regularizer, called patch normalizing flow regularizer (patchNR), involves a normalizing flow learned on small patches of very few images. In particular, the training is independent of the considered inverse problem such that the same regularizer can be applied for different forward operators acting on the same class of images. By investigating the distribution of patches versus those of the whole image class, we prove that our model is indeed a MAP approach. Numerical examples for low-dose and limited-angle computed tomography (CT) as well as superresolution of material images demonstrate that our method provides very high quality results. The training set consists of just six images for CT and one image for superresolution. Finally, we combine our patchNR with ideas from internal learning for performing superresolution of natural images directly from the low-resolution observation without knowledge of any high-resolution image.

show abstract

“…In particular, the authors of [71] proposed the negative log likelihood of all patches of an image as a regularizer, where the underlying patch distribution was assumed to follow a Gaussian mixture model (GMM) which parameters were learned from few clean images. The arising method is still competitive to many approaches based on deep learning and several improvements and extensions were suggested recently [16,49,57]. However, even though GMMs can approximate any probability density function if the number of components is large enough, they suffer from limited flexibility in case of a fixed number of components, see [17] and the references therein.…”

Section: Introductionmentioning

confidence: 99%

PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization

Altekrüger¹,

Denker²,

Hagemann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning neural networks using only a small amount of data is an important ongoing research topic with tremendous potential for applications. In this paper, we introduce a regularizer for the variational modeling of inverse problems in imaging based on normalizing flows. Our regularizer, called patchNR, involves a normalizing flow learned on patches of very few images. The subsequent reconstruction method is completely unsupervised and the same regularizer can be used for different forward operators acting on the same class of images. By investigating the distribution of patches versus those of the whole image class, we prove that our variational model is indeed a MAP approach. Our model can be generalized to conditional patchNRs, if additional supervised information is available. Numerical examples for low-dose CT, limited-angle CT and superresolution of material images demonstrate that our method provides high quality results among unsupervised methods, but requires only few data.

show abstract

Posterior Sampling for Image Restoration using Explicit Patch Priors

Cited by 3 publications

References 30 publications

A Theory of the Distortion-Perception Tradeoff in Wasserstein Space

A Theory of the Distortion-Perception Tradeoff in Wasserstein Space

PatchNR: learning from very few images by patch normalizing flow regularization

PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization

Contact Info

Product

Resources

About