Deep Learning for Inverse Problems: Bounds and Regularizers

Amjad, Jaweria; Lyu, Zhaoyan; Rodrigues, Miguel R. D.

doi:10.48550/arxiv.1901.11352

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regularization with an Extra Penalty Some works have tried to maintain the weight constraints using an additional penalty on the objective function, which can be viewed as a regularization. This regularization technique is mainly used for learning the weight matrices with orthogonality constraints, for its efficiency in computation [4], [16], [146], [154], [155]. Orthogonal regularization methods have demonstrated improved performance in image classification [16], [154], [156], [157], resisting attacks from adversarial examples [158], neural photo editing [159] and training GANs [23], [30].…”

Section: Training With Constraintsmentioning

confidence: 99%

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Huang¹,

Qin²,

Zhou³

et al. 2020

Preprint

View full text Add to dashboard Cite

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.

show abstract

Section: Training With Constraintsmentioning

confidence: 99%

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Huang¹,

Qin²,

Zhou³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the initial orthogonality can be broken down and is not necessarily sustained throughout training [71]. Previous works have tried to maintain the orthogonal weight matrix by imposing an additional orthogonality penalty on the objective function, which can be viewed as a 'soft orthogonal constraint' [51,66,71,5,3]. These methods show improved performance in image classification [71,76,40,5], resisting attacks from adversarial examples [13], neural photo editing [11] and training generative adversarial networks (GAN) [10,47].…”

Section: Introductionmentioning

confidence: 99%

Controllable Orthogonalization in Training DNNs

Huang

Liu

Zhu

et al. 2020

Preprint

View full text Add to dashboard Cite

Orthogonality is widely used for training deep neural networks (DNNs) due to its ability to maintain all singular values of the Jacobian close to 1 and reduce redundancy in representation. This paper proposes a computationally efficient and numerically stable orthogonalization method using Newton's iteration (ONI), to learn a layer-wise orthogonal weight matrix in DNNs. ONI works by iteratively stretching the singular values of a weight matrix towards 1. This property enables it to control the orthogonality of a weight matrix by its number of iterations. We show that our method improves the performance of image classification networks by effectively controlling the orthogonality to provide an optimal tradeoff between optimization benefits and representational capacity reduction. We also show that ONI stabilizes the training of generative adversarial networks (GANs) by maintaining the Lipschitz continuity of a network, similar to spectral normalization (SN), and further outperforms SN by providing controllable orthogonality.

show abstract

Deep Learning for Inverse Problems: Bounds and Regularizers

Cited by 2 publications

References 0 publications

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Controllable Orthogonalization in Training DNNs

Contact Info

Product

Resources

About