Convergence of Parameter Estimates for Regularized Mixed Linear Regression Models

Abstract

We consider Mixed Linear Regression (MLR), where training data have been generated from a mixture of distinct linear models (or clusters) and we seek to identify the corresponding coefficient vectors. We introduce a Mixed Integer Programming (MIP) formulation for MLR subject to regularization constraints on the coefficient vectors. We establish that as the number of training samples grows large, the MIP solution converges to the true coefficient vectors in the absence of noise. Subject to slightly stronger assumptions, we also establish that the MIP identifies the clusters from which the training samples were generated. In the special case where training data come from a single cluster, we establish that the corresponding MIP yields a solution that converges to the true coefficient vector even when training data are perturbed by (martingale difference) noise. We provide a counterexample indicating that in the presence of noise, the MIP may fail to produce the true coefficient vectors for more than one clusters. We also provide numerical results testing the MIP solutions in synthetic examples with noise.

Compound identification using penalized linear regression.

“…We now investigate the properties of (21). Specifically, we prove the following assertion: the ODEs ( 21 From (21b), we have…”

“…will converge to the true parameters β * 1 , β * 2 ∈ R d when n → ∞ as shown by Wang et al [21]. However, it is hard in general to find a computational algorithm for solving this mixed optimization problem.…”

“…By (21) and the fact that the asymptotically stationary p.d.f of ϕ k has the rotation invariant property, we have G = c 0 I with c 0 being a positive constant [44]. From (26), it follows that ∥R(t)−G∥ ≤ ∥R(0)−G∥.…”

“…Moreover, we give some illustrations on properties of f (β(t)). By (21) and Assumptions 2-3, we have…”

“…In the optimization-based method, minimizing the non-convex mean square error of the MLR problem can be converted into the optimization of some objective functions with nice properties such as convexity or smoothness [16]. However, solving the optimization problem is essentially to optimize a nuclear norm function with linear constraints [20] or to solve a mixed integer programming problem [21], both of which may lead to high computational cost. The EM algorithm [17], including E-step and M-step, is a general technique to estimate unknown parameters with hidden random variables.…”