INV-ASKIT: A Parallel Fast Direct Solver for Kernel Matrices

Abstract

We present a parallel algorithm for computing the approximate factorization of an N -by-N kernel matrix. Once this factorization has been constructed (with N log 2 N work), we can solve linear systems with this matrix with N log N work. Kernel matrices represent pairwise interactions of points in metric spaces. They appear in machine learning, approximation theory, and computational physics. Kernel matrices are typically dense (matrix multiplication scales quadratically with N ) and ill-conditioned (solves can require 100s of Krylov iterations). Thus, fast algorithms for matrix multiplication and factorization are critical for scalability.Recently we introduced ASKIT, a new method for approximating a kernel matrix that resembles N-body methods.Here we introduce INV-ASKIT, a factorization scheme based on ASKIT. We describe the new method, derive complexity estimates, and conduct an empirical study of its accuracy and scalability. We report results on realworld datasets including "COVTYPE" (0.5M points in 54 dimensions), "SUSY" (4.5M points in 8 dimensions) and "MNIST" (2M points in 784 dimensions) using shared and distributed memory parallelism. In our largest run we approximately factorize a dense matrix of size 32M × 32M (generated from points in 64 dimensions) on 4,096 Sandy-Bridge cores. To our knowledge these results improve the state of the art by several orders of magnitude.

An N log N Parallel Fast Direct Solver for Kernel Matrices

“…We also briefly summarize the ASKIT algorithm which we use as the basis for our new methods. We describe parallel factorization schemes in §II-B and highlight the novelty of our approach over [36]. We then introduce our hybrid iterative/direct solver in §II-C.…”

“…This algorithm improves on the one in [36] by removing the extra subtree traversals that result in O(N log 2 N ) complexity. Instead, our algorithm exploits the nested structure ofP α α resulting in an N log N complexity for the factorization.…”

“…Parallel direct solver. The parallelization is essentially identical to the scheme proposed in [36]. Each subtree (a set of points {x}) is assigned to a distributed-memory process (or a worker).…”

“…Also our methods cannot be applied to hierarchical decompositions in which D is sparse and not just block diagonal. For such decompositions, our method can be used as a preconditioner, as discussed in [36].…”

A Study of Clustering Techniques and Hierarchical Matrix Formats for Kernel Ridge Regression

“…A possible future work is to fully compare the classification accuracy and the speed with STRUMPACK to that obtained from the method described in [4].…”