Complex best r-term approximations almost always exist in finite dimensions

Abstract

We show that in finite-dimensional nonlinear approximations, the best r-term approximant of a function f almost always exists over C but that the same is not true over R, i.e., the infimum inf f 1 ,...,fr ∈Y f − f1 − · · · − fr is almost always attainable by complex-valued functions f1, . . . , fr in Y , a set of functions that have some desired structures. Our result extends to functions that possess special properties like symmetry or skew-symmetry under permutations of arguments. For the case where Y is the set of separable functions, the problem becomes that of best rank-r tensor approximations. We show that over C, any tensor almost always has a unique best rank-r approximation. This extends to other notions of tensor ranks such as symmetric rank and alternating rank, to best r-block-terms approximations, and to best approximations by tensor networks. When applied to sparse-plus-low-rank approximations, we obtain that for any given r and k, a general tensor has a unique best approximation by a sum of a rank-r tensor and a k-sparse tensor with a fixed sparsity pattern; this arises in, for example, estimation of covariance matrices of a Gaussian hidden variable model with k observed variables conditionally independent given r hidden variables. The existential (but not the uniqueness) part of our result also applies to best approximations by a sum of a rank-r tensor and a k-sparse tensor with no fixed sparsity pattern, as well as to tensor completion problems.2010 Mathematics Subject Classification. 15A69, 41A50, 41A52, 41A65, 51M35.

Best k-Layer Neural Network Approximations

“…This article studies the best k-layer neural network approximation from the perspective of our earlier work [15], where we studied similar issues for the best k-term approximation. An important departure from [15] is that a neural network is not an algebraic object because the most common activation functions σ max , σ tanh , σ exp are not polynomials; thus the algebraic techniques in [15] do not apply in our study here and are relevant at best only through analogy.…”

“…Our article seeks to address this gap. The geometry of the set in (11) will play an important role in studying these problems, much like the role played by the geometry of rank-k tensors in [15].…”

“…An example of a join locus is the set of "sparse-plus-low-rank" matrices [15,Section 8.1]; an example of a rth secant locus is the set of rank-r tensors [15,Section 7].…”

“…There is a parallel with [15], where we applied methods from algebraic and differential geometry to study the empirical risk minimization problem corresponding to nonlinear approximation, i.e., where one seeks to approximate a target function by a sum of k atoms ϕ 1 , . .…”

A Note on Nonclosed Tensor Formats

“…To be precise, this result holds for R as underlying field. A new result by Qi-Michałek-Lim [13] states that in the complex case this exceptional set is of measure zero. Many numerical approaches lead to optimisation problems within the set R r .…”