The sum of nonsingular matrices is often nonsingular

Solymosi, József

doi:10.1016/j.laa.2018.04.019

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If X and V are nonsingular, then their product, Y = XV , is also non-singular. [29]) Let Z ∈ R n×n be the set of non-singular matrices. Thus, for many pairs X1, X2 ∈ Z, the sum Y = X1 + X2 is non-singular.…”

Section: Theoretical Study Of Dnns With Skip Connections and Concatenated Hidden Represenationsmentioning

confidence: 99%

Why Do Deep Neural Networks with Skip Connections and Concatenated Hidden Representations Work?

Oyedotun

Aouada

2020

Neural Information Processing

View full text Add to dashboard Cite

Training the classical-vanilla deep neural networks (DNNs) with several layers is problematic due to optimization problems. Interestingly, skip connections of various forms (e.g. that perform the summation or concatenation of hidden representations or layer outputs) have been shown to allow the successful training of very DNNs. Although there are ongoing theoretical works to understand very DNNs that employ the summation of the outputs of different layers (e.g. as in the residual network), there is none to the best of our knowledge that has studied why DNNs that concatenate of the outputs of different layers (e.g. as seen in Inception, FractalNet and DenseNet) works. As such, we present in this paper, the first theoretical analysis of very DNNs with concatenated hidden representations based on a general framework that can be extended to specific cases. Our results reveal that DNNs with concatenated hidden representations circumnavigate the singularity of hidden representation, which is catastrophic for optimization. For substantiating the theoretical results, extensive experiments are reported on standard datasets such as the MNIST and CIFAR-10.

show abstract

Section: Theoretical Study Of Dnns With Skip Connections and Concatenated Hidden Represenationsmentioning

confidence: 99%

Why Do Deep Neural Networks with Skip Connections and Concatenated Hidden Representations Work?

Oyedotun

Aouada

2020

Neural Information Processing

View full text Add to dashboard Cite

show abstract

“…The main technical tool will be a lemma of Croot-Lev-Pach [Croot et al, 2017] that was the main ingredient in the recent solution of the cap-set problem [Ellenberg and Gijswijt, 2017] and has found many other applications since then (e.g., [Green, 2016, Solymosi, 2018, Dvir and Edelman, 2017, Fox and Lovász, 2017 to name a few).…”

Section: Interpolation Degreementioning

confidence: 99%

A Sauer-Shelah-Perles Lemma for Sumsets

Dvir¹,

Moran

2018

Electron. J. Combin.

View full text Add to dashboard Cite

We extend the Sauer-Shelah-Perles lemma to an abstract setting that is formalized using the language of lattices. Our extension applies to all finite lattices with nonvanishing Möbius function, a rich class of lattices which includes all geometric lattices (or matroids) as a special case.For example, our extension implies the following result in Algebraic Combinatorics: let F be a family of subspaces of F n q . We say that F shatters a subspace U if for every subspace U ′ ≤ U there is F ∈ F such that F ∩U = U ′ . Then, if |F| > n 0 q +· · ·+ n d q then F shatters some (d+1)-dimensional subspace (where n k q denotes the number of k-dimensional subspaces in

show abstract

The sum of nonsingular matrices is often nonsingular

Abstract: If M is a set of nonsingular k × k matrices then for many pairs of matrices, A, B ∈ M, the sum is nonsingular, det(A + B) = 0. We prove a more general statement on nonsingular sums with a geometric application.

Cited by 2 publications

References 9 publications

Why Do Deep Neural Networks with Skip Connections and Concatenated Hidden Representations Work?

Why Do Deep Neural Networks with Skip Connections and Concatenated Hidden Representations Work?

A Sauer-Shelah-Perles Lemma for Sumsets

Contact Info

Product

Resources

About