Statistical properties of sketching algorithms

Abstract

Sketching is a probabilistic data compression technique that has been largely developed in the computer science community. Numerical operations on big datasets can be intolerably slow; sketching algorithms address this issue by generating a smaller surrogate dataset. Typically, inference proceeds on the compressed dataset. Sketching algorithms generally use random projections to compress the original dataset and this stochastic generation process makes them amenable to statistical analysis. We argue that the sketched data can be modelled as a random sample, thus placing this family of data compression methods firmly within an inferential framework. In particular, we focus on the Gaussian, Hadamard and Clarkson-Woodruff sketches, and their use in single pass sketching algorithms for linear regression with huge n. We explore the statistical properties of sketched regression algorithms and derive new distributional results for a large class of sketched estimators. A key result is a conditional central limit theorem for data oblivious sketches. An important finding is that the best choice of sketching algorithm in terms of mean square error is related to the signal to noise ratio in the source dataset. Finally, we demonstrate the theory and the limits of its applicability on two real datasets.

Sketching in Bayesian High Dimensional Regression With Big Data Using Gaussian Scale Mixture Priors

“…(D) puts restriction on the smallest eigenvalue of the matrix X n,ξ Xn,ξ /m n . Notably, Gaussian sketching approximately preserves the isometry condition (Ahfock et al, 2017), so that ∃ η 0 > 0 with the property that e min ( X n,ξ Xn,ξ /m n ) ≥ η 0 e min (X n,ξ X n,ξ /n) with probability f n depending on m n and p n . This, together with assumption A 1 (3) in Song and Liang (2017) is used to argue that assumption (D) is satisfied with a positive probability.…”

“…where the inequality in the fourth line follows from the fact that there exists η > 0 such that || Xn v|| 2 2 ≤ η||X n v|| 2 2 , for all v (Ahfock et al, 2017). This implies that e max ( X n Xn ) = sup…”

“…First, by Lemma 7.1, e min ((Φ n Φ n ) −1 ) ≥ n/( √ n + √ m n + o( √ n)) 2 almost surely. Second, e min ( X n,ξ Xn,ξ /m n ) ≥ e min (X n,ξ X n,ξ /n)η, for some η > 0, by Ahfock et al (2017). Thus, using Assumption (D), it follows that e min ( X n,ξ Xn,ξ /m n ) ≥ C4 , for some constant C4 > 0 and for all ξ ⊃ ξ * such that |ξ| ≤ s n + sn .…”

“…Chowdhury et al (2018) propose a data-dependent algorithm in light of the ridge leverage scores. Other related works include Ailon and Chazelle (2006); Drineas et al (2011); Raskutti and Mahoney (2016); Ahfock et al (2017); Huang (2018). To the best of our knowledge, we are the first to offer efficient and principled Bayesian computation algorithm with linear regressions involving large n and p using data sketching.…”

Randomized Algorithms for Computation of Tucker decomposition and Higher Order SVD (HOSVD)

“…This procedure can be performed by multiplying a given matrix by a random matrix from the right-hand or left-hand side. This is called random projection and it has been shown that this procedure preserves the Euclidean distance among the points approximately [78], [79]. To be precise, let us consider matrix X and target rank R In the first stage of the random projection approach, we generate a random matrix Ω = [ω 1 , ω 2 , .…”