Stable recovery of entangled weights: Towards robust identification of deep neural networks from minimal samples

Finite Sample Identification of Wide Shallow Neural Networks with Biases

“…Making these passages rigorous is beyond the scope of this work, and we leave it as an open question. We also highlight that assumption (M3) is common in the related literature [3,22,23,25].…”

“…This last quantity is provably smaller than the error (22) at initialization, see the discussion after Proposition 1. In the worst case, when all weight errors are aligned, ∆ W,1 is dominated by…”

“…It is natural to compare the residual error term ∆ W,1 after gradient descent with the error on the shifts before gradient descent, i.e., at initialization as given by Proposition 1 (cf. (22)). If we assume randomness on the weight errors (with variance matching the upper bound in (11)), i.e., ŵπ(k) − w k ∼ i.i.d.…”

“…, w m ⊗ w m } and, hence, multiple samples of independent Hessians allow to compute an approximating subspace W ≈ W. The construction of such a subspace is based exclusively on second order information and differs from the tensor approach in [30], which uses higher order tensor decomposition. The identification of the weights is then performed by projected gradient ascent, the so-called subspace power method [22,32,33], seeking for solutions of…”

“…However, while [25] require the weights to be linearly independent (and, thus, m ≤ D), we tackle the challenging overcomplete case m > D. Furthermore, for the identification of the weights we use (2), namely a robust non-linear program over vectors, which is significantly less computationally expensive than the minimum rank selection of [25]. Our analysis improves upon [22] by allowing to go beyond a linear scaling between m and D, and it takes advantage of the new insights provided by [32] on the subspace power method.…”