Non-Euclidean Contractivity of Recurrent Neural Networks

Abstract

Critical questions in neuroscience and machine learning can be addressed by establishing strong stability, robustness, entrainment, and computational efficiency properties of neural network models. The usefulness of such strong properties motivates the development of a comprehensive contractivity theory for neural networks.This paper makes two sets of contributions. First, we develop novel general results on non-Euclidean matrix measures and nonsmooth contraction theory. Regarding 1/ ∞ matrix measures, we show their quasiconvexity with respect to positive diagonal weights, their monotonicity with respect to principal submatrices, and provide closed form expressions for certain matrix polytopes. These results motivate the introduction of M -Hurwitz matrices, i.e., matrices whose Metzler majorant is Hurwitz. Regarding nonsmooth contraction theory, we show that the one-sided Lipschitz constant of a Lipschitz vector field is equal to the essential supremum of the matrix measure of its Jacobian. Second, we apply these general results to classes of recurrent neural circuits, including Hopfield, firing rate, Persidskii, Lur'e and other models. For each model, we compute the optimal contraction rate and weighted non-Euclidean norm via a linear program or, in some special cases, via an M -Hurwitz condition on the synaptic matrix. Our analysis establishes also absolute contraction and total contraction.

Contraction Analysis of Hopfield Neural Networks with Hebbian Learning

“…In summary, we have shown that the solution of ( 8) is equal to x(t) = q − c lin t for all t ∈ [0, t c ] and is equal to d at time t c . Therefore from (9) and being x(t c ) = q − c lin t c , for all time t > t c we have x(t) = q − c lin t c e −cexp(t−tc) . Specifically, x(t) > 0 for all t > t c , thus it can never be the case x(t) < −d.…”

“…In [9,Theorem 16] condition ( 5) is generalized for locally Lipschitz function, for which, by Rademacher's theorem, the Jacobian exists almost everywhere (a.e.) in C. Specifically, if f is locally Lipschitz, then f is infinitesimally contracting on C if condition (5) holds a.e.…”