Learning Mixtures of Sparse Linear Regressions Using Sparse Graph Codes

Abstract

In this paper, we consider the mixture of sparse linear regressions model. Let β (1) , . . . , β (L) ∈ C n be L unknown sparse parameter vectors with a total of K non-zero elements. Noisy linear measurements are obtained in the form yi = x H i β ( i ) + wi, each of which is generated randomly from one of the sparse vectors with the label i unknown. The goal is to estimate the parameter vectors efficiently with low sample and computational costs. This problem presents significant challenges as one needs to simultaneously solve the demixing problem of recovering the labels i as well as the estimation problem of recovering the sparse vectors β ( ) .Our solution to the problem leverages the connection between modern coding theory and statistical inference. We introduce a new algorithm, Mixed-Coloring, which samples the mixture strategically using query vectors xi constructed based on ideas from sparse graph codes. Our novel code design allows for both efficient demixing and parameter estimation. To find K non-zero elements, it is clear that we need at least Θ(K) measurements, and thus the time complexity is at least Θ(K). In the noiseless setting, for a constant number of sparse parameter vectors, our algorithm achieves the order-optimal sample and time complexities of Θ(K). In the presence of Gaussian noise, 1 for the problem with two parameter vectors (i.e., L = 2), we show that the Robust Mixed-Coloring algorithm achieves near-optimal Θ(K polylog(n)) sample and time complexities. When K = O(n α ) for some constant α ∈ (0, 1) (i.e., K is sublinear in n), we can achieve sample and time complexities both sublinear in the ambient dimension. In one of our experiments, to recover a mixture of two regressions with dimension n = 500 and sparsity K = 50, our algorithm is more than 300 times faster than EM algorithm, with about one third of its sample cost.

Support Recovery of Sparse Signals from a Mixture of Linear Measurements

“…The most relevant works in this setting would be [44], [27] and [29], all of which were concerned with approximately recovering the k-sparse unknown vectors v 1 , v 2 , . .…”

“…While approximate recovery of vectors can also be translated into support recovery, the results of [44] and [27] are valid only under the restrictive assumption that the sparse vectors all belong to some scaled integer lattice. The result of [29] does not have any restriction, but it holds only when ℓ = 2.…”

“…In a recent line of work that started with [44], a generalization of the sparse recovery problem is considered [27,29,19,9], where instead of one sparse vector, multiple unknown sparse vectors are to be recovered. However any attempt to obtain a measurement (linear or 1-bit) from the vectors results in a mixture model, where a vector from the unknown set is picked uniformly to generate the response.…”

“…The mixture of sparse recovery models of [44] and followup works can be framed as a Mixture of Linear Classifiers (MLC) or a Mixture of Linear Regressions (MLR) problems. The statistical model in MLC is the following: there exists ℓ unknown hyperplanes with normal vectors v 1 , v 2 , .…”

“…In this work, we tackle the problem of support recovery of sparse vectors for both MLR and MLC model in the active query based setting of [44,27,29,19]. Our goal is to recover the support of all the unknown sparse vectors (hyperplane normals) with minimum number of measurements.…”