Deterministic Sparse Column Based Matrix Reconstruction via Greedy Approximation of SVD

Abstract

Abstract. Given a matrix A ∈ R m×n of rank r, and an integer k < r, the top k singular vectors provide the best rank-k approximation to A. When the columns of A have specific meaning, it is desirable to find (provably) "good" approximations to A k which use only a small number of columns in A. Proposed solutions to this problem have thus far focused on randomized algorithms. Our main result is a simple greedy deterministic algorithm with guarantees on the performance and the number of columns chosen. Specifically, our greedy algorithm chooses c columns from A with c = Owhere Cgr is the matrix composed of the c columns, C + gr is the pseudoinverse of Cgr (CgrC + gr A is the best reconstruction of A from Cgr), and µ(A) is a measure of the coherence in the normalized columns of A. The running time of the algorithm is O(SV D(A k ) + mnc) where SV D(A k ) is the running time complexity of computing the first k singular vectors of A. To the best of our knowledge, this is the first deterministic algorithm with performance guarantees on the number of columns and a (1 + ) approximation ratio in Frobenius norm. The algorithm is quite simple and intuitive and is obtained by combining a generalization of the well known sparse approximation problem from information theory with an existence result on the possibility of sparse approximation. Tightening the analysis along either of these two dimensions would yield improved results.

Randomized Algorithms for Matrices and Data

“…For this and other reasons, a common task in genetics and other areas of data analysis is the following: given an input data matrix A and a parameter k, find the best subset of exactly k actual DNA SNPs or actual genes, i.e., actual columns or rows from A, to use to cluster individuals, reconstruct biochemical pathways, reconstruct signal, perform classification or inference, etc. Unfortunately, common formalizations of this algorithmic problem-including looking for the k actual columns that capture the largest amount of information or variance in the data or that are maximally uncorrelated-lead to intractable optimization problems [22,23]. For example, consider the so-called Column Subset Selection Problem [24]: given as input an arbitrary m × n matrix A and a rank parameter k, choose the set of exactly k columns of A s.t.…”

“…That being said, running the risk of such a failure might be acceptable if one can efficiently couple to a diagnostic to check for such a failure, and if one can then correct for it by choosing more samples if necessary. The best numerical implementations of randomized matrix algorithms for low-rank matrix approximation do just this, and the strongest results in terms of minimizing p take advantage of Condition (22) in a somewhat different way than was originally used in the analysis of the CSSP [14]. For example, rather than choosing O(k log k) dimensions and then filtering them through exactly k dimensions, as the relative-error random sampling and relative-error random projection algorithms do, one can choose some number ℓ of dimensions and project onto a k ′ -dimensional subspace, where k < k ′ ≤ ℓ, while exploiting Condition (22) to bound the error, as appropriate for the computational environment at hand [14].…”

“…25 Of course, there is randomization inside 22 Stating this in terms of the singular vectors is a convenience, but it can create confusion. In particular, although computing the SVD is sufficient, it is by no means necessary-here, U can be any orthogonal matrix spanning the column space of A [21].…”

“…35 Of course, this should not be completely unexpected, given that Condition (22) shows that the behavior of algorithms depends on the interaction between different subspaces associated with the input matrix A. When stronger assumptions are made about the data, stronger bounds can often be obtained.…”

Greedy column subset selection for large-scale data sets

“…In this implementation, the MATLAB qr function is first used to calculate the QR decomposition with column pivoting and then the columns are swapped using the criterion specified by Gu and Eisenstat [31]. 8 -ApproxSVD: is the sparse approximation of Singular Value Decomposition (SVD) [9,10]. The algorithm was implemented in MATLAB.…”