Quantizing Heavy-Tailed Data in Statistical Estimation: (Near) Minimax Rates, Covariate Quantization, and Uniform Recovery

Abstract

Modern datasets often exhibit heavy-tailed behavior, while quantization is inevitable in digital signal processing and many machine learning problems. This paper studies the quantization of heavy-tailed data in several fundamental statistical estimation problems where the underlying distributions have bounded moments of some order (no greater than 4). We propose to truncate and properly dither the data prior to a uniform quantization. Our major standpoint is that (near) minimax rates of estimation error could be achieved by computationally tractable estimators based on the quantized data produced by the proposed scheme. In particular, concrete results are worked out for covariance estimation, compressed sensing (also interpreted as sparse linear regression), and matrix completion, all agreeing that the quantization only slightly worsens the multiplicative factor. Additionally, while prior results focused on the quantization of responses (i.e., measurements), we study compressed sensing where the covariates (i.e., sensing vectors) are also quantized; in this case, though our recovery program is non-convex (since our covariance matrix estimator lacks positive semi-definiteness), we prove that all local minimizers enjoy near-optimal estimation error. Moreover, by the concentration inequality of the product process and a covering argument, we establish a near minimax uniform recovery guarantee for quantized compressed sensing with heavy-tailed noise. Finally, numerical simulations are provided to corroborate our theoretical results.

Quantized Low-Rank Multivariate Regression With Random Dithering

“…There has been rapidly growing literature on quantized compressed sensing [5], [14], [15], [20], [22], [47], [57], [59], [66], quantized matrix completion [4], [10], [14], [15], [19], [35], and more recently quantized covariance estimation [14], [15], [21], but we are not aware of any earlier work on quantized LRMR (or more generally put, quantized multiresponse regression). Closest to this paper are prior developments on compressed sensing (CS) under dithered uniform quantization [14], [57], [59], [66], which we briefly review here. Recall that the (noiseless) CS problem is to recover a structured (e.g., sparse/low-rank) signal θ 0 ∈ R d from the data of…”

“…In [57], Sun et al extended [59] to corrupted sensing that aims at separating signal and corruption. While [57], [59], [66] only considered the quantization of y k , a recent work [14] developed the quantization method for x k , i.e., via the same dithered uniform quantizer but with uniform dither substituted with triangular dither.…”

“…Although we adopt a similar dithered quantization scheme (specifically, similar to [14]), the estimation problem in this paper totally differs from CS. In particular, we will study regression models with multivariate response that can be a vector with considerably large dimension (LRMR in section III) or even a huge matrix (see section IV), as in sharp contrast to the scalar measurement y k in CS.…”

“…] . Such triangular dither was also adopted in [14] when studying covariate quantization in compressed sensing.…”

“…Additionally, while sending the quantized gradient is the mainstream in machine learning, it may be inefficient in distributed learning with a huge number of parameters to learn; in this case, transmitting some important quantized data samples could provably reduce the communication cost [28]. Thus, it is of particular interest to theoretically investigate the interplay between parameter learning and data quantization in some fundamental statistical learning or estimation problems, e.g., [14], [15], [21].…”