Low rank approximation with entrywise l ₁ -norm error

Abstract

We study the 1 -low rank approximation problem, where for a given n × d matrix A and approximation factor α ≥ 1, the goal is to output a rank-k matrix A for whichwhere for an n × d matrix C, we let C 1 = n i=1 d j=1 |C i,j |. This error measure is known to be more robust than the Frobenius norm in the presence of outliers and is indicated in models where Gaussian assumptions on the noise may not apply. The problem was shown to be NP-hard by Gillis and Vavasis and a number of heuristics have been proposed. It was asked in multiple places if there are any approximation algorithms.We give the first provable approximation algorithms for 1 -low rank approximation, showing that it is possible to achieve approximation factor α = (log d)·poly(k) in nnz(A)+(n+d) poly(k) time, where nnz(A) denotes the number of non-zero entries of A. If k is constant, we further improve the approximation ratio to O(1) with a poly(nd)-time algorithm. Under the Exponential Time Hypothesis, we show there is no poly(nd)-time algorithm achieving a (1 + 1 log 1+γ (nd) )approximation, for γ > 0 an arbitrarily small constant, even when k = 1.We give a number of additional results for 1 -low rank approximation: nearly tight upper and lower bounds for column subset selection, CUR decompositions, extensions to low rank approximation with respect to p -norms for 1 ≤ p < 2 and earthmover distance, low-communication distributed protocols and low-memory streaming algorithms, algorithms with limited randomness, and bicriteria algorithms. We also give a preliminary empirical evaluation. * Work done while visiting IBM Almaden. arXiv:1611.00898v1 [cs.DS] 3 Nov 2016 1 -low rank approximation can better handle missing data, is appropriate in noise models for which 1 log 1+γ (n) )-approximation, for γ > 0 an arbitrarily small constant, even when k = 1. The latter strengthens the NP-hardness result of [GV15].We also give a number of results for variants of 1 -low rank approximation which are studied for Frobenius norm low rank approxiation; prior to our work nothing was known about these problems.Column Subset Selection and CUR Decomposition: In the column subset selection problem, one seeks a small subset C of columns of A for which there is a matrix X for which CX − A is small, under some norm. The matrix CX provides a low rank approximation to A which is often more interpretable, since it stores actual columns of A, preserves sparsity, etc. These have been extensively studied when the norm is the Frobenius or operator norm (see, e.g., [BMD09, DR10, BDM11] and the references therein). We initiate the study of this problem with respect to the 1 -norm. We first prove an existence result, namely, that there exist matrices A for which any subset C of poly(k) columns satisfies minwhere γ > 0 is an arbitrarily small constant. This result is in stark contrast to the Frobenius norm for which for every matrix there exist O( k ) columns for which the approximation factor is 1 + . We also show that our bound is nearly optimal in this regime, by showing for every matrix t...

A PTAS for ℓp-Low Rank Approximation

“…The solution V to min V SU * V − SA 1,2 is a √ r-approximation to the original problem min V SU * V − SA 1 . In the · 1,2 norm, the solution V can be written in terms of the so-called normal equations for regression, namely, V = (SU * ) † SA, where C † denotes the Moore-Penrose pseudoinverse of C. The key property exploited in [67] is then that although we do not know U * , (SU * ) † SA is a k-dimensional subspace in the row span of SA providing a √ r-approximation, and one does know SA. This line of reasoning ultimately leads to a poly(k)-approximation.…”

“…There is also related work on robust PCA [15,17,55,56,72,74] and measures which minimize the sum of Euclidean norms of rows [20,[23][24][25]65], though neither directly gives an algorithm for 1 -low rank approximation. Song et al [67] gave the first approximation algorithms with provable guarantees for entrywise p -low rank approximation for p ∈ [1,2). Their algorithm provides a poly(k log n) approximation and runs in polynomial time, that is, the algorithm outputs a matrix B for which A − B p ≤ poly(k log n) min rank-k A A−A p .…”

Heuristic Search Algorithm for Dimensionality Reduction Optimally Combining Feature Selection and Feature Extraction

“…In particular, minimizing the entry-wise l 1 or l 0 norms is expected to improve the robustness of the estimation, but unfortunately the problem formulated in these norms turns out to be NP-hard. See, e.g., Gillis (2018), Song (2017), and Bringmann (2017). For the more general case of entrywise l p norms see Chierichetti (2017).…”

On the Complexity of Robust PCA and ℓ1-Norm Low-Rank Matrix Approximation

“…After the publication of an earlier of this paper, Song, Woodruff and Zhong proposed several approximation algorithms for ℓ 1 -LRA [36], addressing the second part of the open question 2 in [38]. In particular, they showed that it is possible to achieve an approximation factor α = (log n) · poly(r) in nnz(M ) + (m + n)poly(r) time, where nnz(M ) denotes the number of non-zero entries of M .…”