Estimating Higher-Order Moments Using Symmetric Tensor Decomposition

Abstract

We consider the problem of decomposing higher-order moment tensors, i.e., the sum of symmetric outer products of data vectors. Such a decomposition can be used to estimate the means in a Gaussian mixture model and for other applications in machine learning. The dth-order empirical moment tensor of a set of p observations of n variables is a symmetric d-way tensor. Our goal is to find a low-rank tensor approximation comprising r ! p symmetric outer products. The challenge is that forming the empirical moment tensors costs Oppn d q operations and Opn d q storage, which may be prohibitively expensive; additionally, the algorithm to compute the low-rank approximation costs Opn d q per iteration. Our contribution is avoiding formation of the moment tensor, computing the low-rank tensor approximation of the moment tensor implicitly using Oppnrq operations per iteration and no extra memory. This advance opens the door to more applications of higher-order moments since they can now be efficiently computed. We present numerical evidence of the computational savings and show an example of estimating the means for higher-order moments.

Subspace power method for symmetric tensor decomposition and generalized PCA

“…Another aspect we would like to study is the possible extension of SPM to implicit settings, as in [62]. Here a low-rank symmetric tensor arises as the population moment of a latent variable model (for example, a mixture of Gaussians).…”

A termination criterion for stochastic gradient descent for binary classification

“…Another common use of GMMs in the literature, similar to our application here, is as test-cases for a learning algorithm intended to solve a more general problem. Examples include clustering; see, e.g., Jiang et al (2019) and Panahi et al (2017) and tensor factorization; see, e.g., Sherman and Kolda (2019).…”