Inverse Problems, Deep Learning, and Symmetry Breaking

Tayal, Kshitij; Lai, Chieh-Hsin; Manekar, Raunak; Zhuang, Zhong; Kumar, Vipin; Sun, Ju

doi:10.48550/arxiv.2003.09077

Cited by 4 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To avoid the difficulty of getting a large set of paired phase and intensity measurements, Zhang et al Zhang et al (2021) used an unpaired dataset for phase-retrieval. Other works of Tayal et al (2020); Houhou et al (2020) have also used deep learning for end-to-end phase recovery. None of these works use the underlying physics in their deep learning approach or they require a large number of training samples; hence, are not directly applicable to the physics applications we are interested in this paper.…”

Section: Related Workmentioning

confidence: 99%

Blaschke Product Neural Networks (BPNN): A Physics-Infused Neural Network for Phase Retrieval of Meromorphic Functions

Dong¹,

Ren²,

Deng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Numerous physical systems are described by ordinary or partial differential equations whose solutions are given by holomorphic or meromorphic functions in the complex domain. In many cases, only the magnitude of these functions are observed on various points on the purely imaginary jω-axis since coherent measurement of their phases is often expensive. However, it is desirable to retrieve the lost phases from the magnitudes when possible. To this end, we propose a physics-infused deep neural network based on the Blaschke products for phase retrieval. Inspired by the Helson and Sarason Theorem, we recover coefficients of a rational function of Blaschke products using a Blaschke Product Neural Network (BPNN), based upon the magnitude observations as input. The resulting rational function is then used for phase retrieval. We compare the BPNN to conventional deep neural networks (NNs) on several phase retrieval problems, comprising both synthetic and contemporary real-world problems (e.g., metamaterials for which data collection requires substantial expertise and is time consuming). On each phase retrieval problem, we compare against a population of conventional NNs of varying size and hyperparameter settings. Even without any hyper-parameter search, we find that BPNNs consistently outperform the population of optimized NNs in scarce data scenarios, and do so despite being much smaller models. The results can in turn be applied to calculate the refractive index of metamaterials, which is an important problem in emerging areas of material science.

show abstract

Section: Related Workmentioning

confidence: 99%

Blaschke Product Neural Networks (BPNN): A Physics-Infused Neural Network for Phase Retrieval of Meromorphic Functions

Dong¹,

Ren²,

Deng³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We approach these issues by considering symmetries of the NN parameterization. Several recent works have brought attention to the importance of symmetries to deep learning (Badrinarayanan et al, 2015 ; Kunin et al, 2020 ; Tayal et al, 2020 ). One important advance is building symmetry aware architectures that automatically generalize, such as convolutional layers granting translation-invariant representations, or many other task-specific symmetries (Liu et al, 2016 ; Barbosa et al, 2021 ; Bertoni et al, 2021 ; Liu and Okatani, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics

Sahs

Pyle

Damaraju

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Understanding the learning dynamics and inductive bias of neural networks (NNs) is hindered by the opacity of the relationship between NN parameters and the function represented. Partially, this is due to symmetries inherent within the NN parameterization, allowing multiple different parameter settings to result in an identical output function, resulting in both an unclear relationship and redundant degrees of freedom. The NN parameterization is invariant under two symmetries: permutation of the neurons and a continuous family of transformations of the scale of weight and bias parameters. We propose taking a quotient with respect to the second symmetry group and reparametrizing ReLU NNs as continuous piecewise linear splines. Using this spline lens, we study learning dynamics in shallow univariate ReLU NNs, finding unexpected insights and explanations for several perplexing phenomena. We develop a surprisingly simple and transparent view of the structure of the loss surface, including its critical and fixed points, Hessian, and Hessian spectrum. We also show that standard weight initializations yield very flat initial functions, and that this flatness, together with overparametrization and the initial weight scale, is responsible for the strength and type of implicit regularization, consistent with previous work. Our implicit regularization results are complementary to recent work, showing that initialization scale critically controls implicit regularization via a kernel-based argument. Overall, removing the weight scale symmetry enables us to prove these results more simply and enables us to prove new results and gain new insights while offering a far more transparent and intuitive picture. Looking forward, our quotiented spline-based approach will extend naturally to the multivariate and deep settings, and alongside the kernel-based view, we believe it will play a foundational role in efforts to understand neural networks. Videos of learning dynamics using a spline-based visualization are available at http://shorturl.at/tFWZ2.

show abstract

“…IR entails estimating an image of interest, denoted as x, from a measurement y = f (x), where f models the measurement process. This model covers classical image processing tasks such as image denoising, superresolution, inpainting, deblurring, and modern computational imaging problems such as MRI/CT reconstruction [8] and phase retrieval [33,40]. In this paper, we assume that f is known.…”

mentioning

confidence: 99%

Self-Validation: Early Stopping for Single-Instance Deep Generative Priors

Li¹,

Zhuang²,

Liang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent works have shown the surprising effectiveness of deep generative models in solving numerous image reconstruction (IR) tasks, even without training data. We call these models, such as deep image prior and deep decoder, collectively as single-instance deep generative priors (SIDGPs). The successes, however, often hinge on appropriate early stopping (ES), which by far has largely been handled in an ad-hoc manner. In this paper, we propose the first principled method for ES when applying SIDGPs to IR, taking advantage of the typical bell trend of the reconstruction quality. In particular, our method is based on collaborative training and self-validation: the primal reconstruction process is monitored by a deep autoencoder, which is trained online with the historic reconstructed images and used to validate the reconstruction quality constantly. Experimentally, on several IR problems and different SIDGPs, our self-validation method is able to reliably detect near-peak performance and signal good ES points. Our code is available at https://sun-umn.github.io/Self-Validation/. IntroductionValidation-based ES is one of the most reliable strategies for controlling generalization errors in supervised learning, especially with potentially overspecified models such as in gradient boosting and modern DNNs [10,28,47]. Beyond supervised learning, ES often remains critical to learning success, but there are no principled ways-universal as validation for supervised learning-to decide when to stop. In this paper, we make the first step toward filling in the gap, and focus on solving IR, a central family of inverse problems, using training-free deep generative models.

show abstract

Inverse Problems, Deep Learning, and Symmetry Breaking

Cited by 4 publications

References 30 publications

Blaschke Product Neural Networks (BPNN): A Physics-Infused Neural Network for Phase Retrieval of Meromorphic Functions

Blaschke Product Neural Networks (BPNN): A Physics-Infused Neural Network for Phase Retrieval of Meromorphic Functions

Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics

Self-Validation: Early Stopping for Single-Instance Deep Generative Priors

Contact Info

Product

Resources

About