Nonlinear Power Method for Computing Eigenvectors of Proximal Operators and Neural Networks

Abstract

Neural networks have revolutionized the field of data science, yielding remarkable solutions in a datadriven manner. For instance, in the field of mathematical imaging, they have surpassed traditional methods based on convex regularization. However, a fundamental theory supporting the practical applications is still in the early stages of development. We take a fresh look at neural networks and examine them via nonlinear eigenvalue analysis. The field of nonlinear spectral theory is still emerging, providing insights about nonlinear operators and systems. In this paper we view a neural network as a complex nonlinear operator and attempt to find its nonlinear eigenvectors. We first discuss the existence of such eigenvectors and analyze the kernel of ReLU networks. Then we study a nonlinear power method for generic nonlinear operators. For proximal operators associated to absolutely one-homogeneous convex regularization functionals, we can prove convergence of the method to an eigenvector of the proximal operator. This motivates us to apply a nonlinear method to networks which are trained to act similarly as a proximal operator. In order to take the nonhomogeneity of neural networks into account we define a modified version of the power method. We perform extensive experiments on various shallow and deep neural networks designed for image denoising. For simple nets, we observe the influence of training data on the eigenvectors. For stateof-the-art denoising networks, we show that eigenvectors can be interpreted as (un)stable modes of the network, when contaminated with noise or other degradations.

Iterative Methods for Computing Eigenvectors of Nonlinear Operators

“…They are inherently quite slow, sometimes hundreds or even thousands of iterations are needed in order to numerically converge. A first analysis of the convergence rate of nonlinear power-methods for onehomogeneous functionals is in [10]. This area surely requires additional focus.…”

“…The last algorithm presented here is related to very general and complex nonlinear operators, which often cannot be expressed analytically. In [23] and [10] the operators considered were nonlinear denoisers, which can be based on classical algorithms or on deep neural networks.…”

“…which for J = T V coincides with the ROF denoising model ( [28]). In [10] it was shown that the process is well defined for a range of parameters α, that the energy is decreasing, J(u k+1 ) ≤ J(u k ), along with a full proof of convergence to a nonlinear eigenvector, in the sense of (6).…”

“…Converged eigenfunction (λ = 1), right, is a highly stable structure for the network. Taken from [10].…”

“…Last but not least, can neural networks benefit from this research field? We have shown in [10] that one can treat an entire neural network (intended for denoising) as a single complex nonlinear operator and find some of its eigenfunctions. They represent highly stable and unstable modes (depending on the eigenvalue).…”