Early Stopping for Deep Image Prior

Wang, Hengkang; Li, Tai–Hui; Zhang, Zhong; Chen, Tiancong; Liang, Hengyue; Sun, Jian

doi:10.48550/arxiv.2112.06074

Cited by 9 publications

(9 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar phenomena have been empirically observed in many other nonconvex problems, where vanilla gradient descent, when coupled with small random initialization (SRI) and early stopping (ES), has good generalization performance even with overpametrization due to the algorithmic regularization effect of SRI annd ES [Woodworth et al, 2020, Ghorbani et al, 2020, Prechelt, 1998, Wang et al, 2021, Li et al, 2018, Stöger and Soltanolkotabi, 2021. This motivates us to study the following question: What is the general behavior of the gradient descent dynamic (GD-M) coupled with SRI and ES?…”

Section: Introductionsupporting

confidence: 59%

Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization

Jiang¹,

Chen²,

Ding³

2022

Preprint

View full text Add to dashboard Cite

We study the asymmetric matrix factorization problem under a natural nonconvex formulation with arbitrary overparamatrization. We consider the model-free setting with no further assumption on the rank or singular values of the observed matrix, where the global optima provably overfit. We show that vanilla gradient descent with small random initialization and early stopping produces the best low-rank approximation of the observed matrix, without any additional regularization. We provide a sharp analysis on relationship between the iteration complexity, initialization size, stepsize and final error. In particular, our complexity bound is almost dimension-free and depends logarithmically on the final error, and our results have lenient requirements on the stepsize and initialization. Our bounds improve upon existing work and show good agreement with numerical experiments.

show abstract

Section: Introductionsupporting

confidence: 59%

Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization

Jiang¹,

Chen²,

Ding³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For the reasons just mentioned, the speed of convergence as a function of performance is essential. Recent works have attempted to study convergence speed from the point of view of the optimizer used, while others have tried to propose ad hoc Early Stopping techniques [25]. In this context, this paper aims to study the interpretability of this model through the various initializations in the literature.…”

Section: Discussionmentioning

confidence: 99%

Deep Image Prior for medical image denoising, a study about parameter initialization

Sapienza

Franchini

Govi

et al. 2022

Front. Appl. Math. Stat.

View full text Add to dashboard Cite

Convolutional Neural Networks are widely known and used architectures in image processing contexts, in particular for medical images. These Deep Learning techniques, known for their ability to extract high-level features, almost always require a labeled dataset, a process that can be computationally expensive. Most of the time in the biomedical context, when images are used they are noisy and the ground-truth is unknown. For this reason, and in the context of Green Artificial Intelligence, recently, an unsupervised method that employs Convolutional Neural Networks, or more precisely autoencoders, has appeared in the panorama of Deep Learning. This technique, called Deep Image Prior (DIP) by the authors, can be used in areas such as denoising, superresolution, and inpainting. Starting from these assumptions, this work analyses the robustness of these networks with respect to different types of initialization. First of all, we analyze the different types of parameters: related to the Batch Norm and the Convolutional layers. For the results, we focus on the speed of convergence and the maximum performance obtained. However, this paper aims to apply acquired information on Computer Tomography noised images. In fact, the final purpose is to test the best initializations of the first phase on a phantom image and then on a real Computer Tomography one. In fact, Computer Tomography together with Magnetic Resonance Imaging and Positron Emission Tomography are some of the diagnostic tools currently available to neuroscientists and oncologists. This work shows how initializations affect final performances and, in addition, how they should be used in the medical image reconstruction field. The section on numerical experiments shows results that on the one hand confirm the importance of a good initialization to obtain fast convergence and high performance; on the other hand, it shows how the method is robust to the processing of different image types: natural and medical. Not a single good initialization is discovered, but many of them could be chosen, according to specific necessities of the single problem.

show abstract

“…DIP often uses early-stopping to deliver a satisfactory reconstruction: the update of θ is stopped early to avoid overfitting to the noise [10]. This has motivated developing automated rules for early stopping [24], [25].…”

Section: Deep Image Priormentioning

confidence: 99%

“…Jo et al [24] propose to penalize the complexity of the reconstruction using Stein's unbiased risk estimator. See also [25] for a stopping criterion based on monitoring the running variance of iterate sequence and references therein for further discussions. [6] suggests at test-time to start optimizing a randomly initialized DIP to match a reconstruction, produced by another method.…”

Section: Deep Image Priormentioning

confidence: 99%

An Educated Warm Start for Deep Image Prior-Based Micro CT Reconstruction

Barbano

Leuschner

Schmidt

et al. 2022

IEEE Trans. Comput. Imaging

View full text Add to dashboard Cite

Deep image prior (DIP) was recently introduced as an effective unsupervised approach for image restoration tasks. DIP represents the image to be recovered as the output of a deep convolutional neural network, and learns the network's parameters such that the model output matches the corrupted observation. Despite its impressive reconstructive properties, the approach is slow when compared to supervisedly learned, or traditional reconstruction techniques. To address the computational challenge, we bestow DIP with a two-stage learning paradigm: (i) perform a supervised pretraining of the network on a simulated dataset; (ii) fine-tune the network's parameters to adapt to the target reconstruction task. We provide a thorough empirical analysis to shed insights into the impacts of pretraining in the context of image reconstruction. We showcase that pretraining considerably speeds up and stabilizes the subsequent reconstruction task from real-measured 2D and 3D micro computed tomography data of biological specimens. The code and additional experimental materials are available at educateddip.github.io/docs.educated_deep_image_prior/.

show abstract

Early Stopping for Deep Image Prior

Cited by 9 publications

References 27 publications

Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization

Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization

Deep Image Prior for medical image denoising, a study about parameter initialization

An Educated Warm Start for Deep Image Prior-Based Micro CT Reconstruction

Contact Info

Product

Resources

About