Geometry of Linear Convolutional Networks

Kohn, Kathlén; Merkh, Thomas; Montúfar, Guido; Trager, Matthew

doi:10.48550/arxiv.2108.01538

Cited by 2 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We already mentioned above the Burer-Monteiro approach to semidefinite programming [9,14], low-rank optimization [22,32,36], computer vision [17], and neural networks [38]. In [47,29], the authors compare the stationary points for (Q) to those of (P) for the case of linear neural networks and prove a special case of our Theorem 2.10 characterizing "1 ⇒ 1". The training of general neural networks, and risk minimization more generally, is naturally given in the form (Q), see [47,Appx.…”

Section: Prior Workmentioning

confidence: 99%

The effect of smooth parametrizations on nonconvex optimization landscapes

Levin¹,

Kileel²,

Boumal³

2022

Preprint

View full text Add to dashboard Cite

We develop new tools to study the landscapes of nonconvex optimization problems. Specifically, we leverage the fact that many such problems can be paired with others via a smooth parametrization of their domains. The global optima of such pairs of problems correspond to each other, but their landscapes can be significantly different. We introduce a framework to relate the two landscapes. Applications include: optimization over low-rank matrices and tensors by optimizing over a factorization; the Burer-Monteiro approach to semidefinite programs; training neural networks by optimizing over their weights and biases; and quotienting out symmetries. In all these examples, one of the two problems is smooth, and hence can be tackled with algorithms for optimization on manifolds. These can find desirable points (e.g., critical points). We determine the properties that ensure these map to desirable points for the other problem via the smooth parametrization. These new tools enable us to strengthen guarantees for an array of optimization problems, previously obtained on a case-bycase basis in the literature.

show abstract

Section: Prior Workmentioning

confidence: 99%

The effect of smooth parametrizations on nonconvex optimization landscapes

Levin¹,

Kileel²,

Boumal³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Similarly to the tensor case, this implies that proofs of guarantees for training algorithms must exploit the structure in specific cost functions (the loss). Additional study of lifts defined by linear neural networks was done in [32,54], where the authors characterize (in our terminology) "1 ⇒1" for lifts defined by linear and linear convolutional architectures.…”

Section: Neural Networkmentioning

confidence: 99%

The effect of smooth parametrizations on nonconvex optimization landscapes

Levin,

Kileel,

Boumal

2024

Math. Program.

View full text Add to dashboard Cite

We develop new tools to study landscapes in nonconvex optimization. Given one optimization problem, we pair it with another by smoothly parametrizing the domain. This is either for practical purposes (e.g., to use smooth optimization algorithms with good guarantees) or for theoretical purposes (e.g., to reveal that the landscape satisfies a strict saddle property). In both cases, the central question is: how do the landscapes of the two problems relate? More precisely: how do desirable points such as local minima and critical points in one problem relate to those in the other problem? A key finding in this paper is that these relations are often determined by the parametrization itself, and are almost entirely independent of the cost function. Accordingly, we introduce a general framework to study parametrizations by their effect on landscapes. The framework enables us to obtain new guarantees for an array of problems, some of which were previously treated on a case-by-case basis in the literature. Applications include: optimizing low-rank matrices and tensors through factorizations; solving semidefinite programs via the Burer–Monteiro approach; training neural networks by optimizing their weights and biases; and quotienting out symmetries.

show abstract

Geometry of Linear Convolutional Networks

Cited by 2 publications

References 15 publications

The effect of smooth parametrizations on nonconvex optimization landscapes

The effect of smooth parametrizations on nonconvex optimization landscapes

The effect of smooth parametrizations on nonconvex optimization landscapes

Contact Info

Product

Resources

About