Inexact Generalized Gauss–Newton for Scaling the Canonical Polyadic Decomposition With Non-Least-Squares Cost Functions

Abstract

The canonical polyadic decomposition (CPD) allows one to extract compact and interpretable representations of tensors. Several optimization-based methods exist to fit the CPD of a tensor for the standard least-squares (LS) cost function. Extensions have been proposed for more general cost functions such as β-divergences as well. For these non-LS cost functions, a generalized Gauss-Newton (GGN) method has been developed. This is a second-order method that uses an approximation of the Hessian of the cost function to determine the next iterate and with this algorithm, fast convergence can be achieved close to the solution. Unfortunately, for large tensors, the exact GGN approach becomes too expensive. In this paper, we therefore propose an inexact GGN method. The approximation of the Hessian is only used implicitly, which greatly improves the scalability of the method. Next, we show that by using a compressed instance of the GGN Hessian approximation, the computation time of the inexact GGN method can be lowered even more, with only limited influence on the convergence speed. Further, the maximum likelihood estimator for Rician distributed data is examined in detail as an example of an alternative cost function. This cost function is useful for the analysis of the moduli of complex data, as in functional magnetic resonance imaging, for instance. Finally, we compare the proposed method to the existing CPD methods and demonstrate the method's speed and effectiveness on synthetic and simulated real-life data.

Regression and Classification With Spline-Based Separable Expansions

“…Even though the GN algorithm is derived for least-squares problems, it can be generalized easily to accommodate other loss functions. Following general results in Schraudolph (2002) and for tensor decompositions (Vandecapelle et al, 2020;Vandecappelle et al, 2021), the generalized GN algorithm can be derived similarly to the strategy in Subsection 3.2. In the dogleg trust-region framework, the necessary GN direction p (d) r is derived starting from the linear system:…”

“…In comparison to prior work which used alternating least squares (ALS) scheme (Beylkin et al, 2009;Garcke, 2010) as means to train the model parameters, the GN algorithm is known to exhibit superior convergence properties (see e.g., Sorber et al, 2013;Vervliet and De Lathauwer, 2019). Building on the results for the GNbased computation of a CPD using alternative cost functions (Vandecappelle et al, 2021), we show that our algorithm can be altered to accommodate logistic cost functions which are more suitable for classification problems.…”

CPD-Structured Multivariate Polynomial Optimization

“…Many data science problems such as latent factor analysis have been solved by reformulating them as tensor decomposition problems [9][10][11][12]. An inexact Gauss-Newton algorithm has been proposed for scaling the CPD of large tensors with non-least-squares cost functions [13]. Moreover, generalized Gauss-Newton algorithm with its efficient parallel implementation has been proposed for tensor completion with generalized loss functions [14].…”