A Pseudo-Likelihood Approach to Linear Regression with Partially Shuffled Data

Abstract

Recently, there has been significant interest in linear regression in the situation where predictors and responses are not observed in matching pairs corresponding to the same statistical unit as a consequence of separate data collection and uncertainty in data integration. Mismatched pairs can considerably impact the model fit and disrupt the estimation of regression parameters. In this paper, we present a method to adjust for such mismatches under "partial shuffling" in which a sufficiently large fraction of (predictors, response)-pairs are observed in their correct correspondence. The proposed approach is based on a pseudo-likelihood in which each term takes the form of a two-component mixture density. Expectation-Maximization schemes are proposed for optimization, which (i) scale favorably in the number of samples, and (ii) achieve excellent statistical performance relative to an oracle that has access to the correct pairings as certified by simulations and case studies. In particular, the proposed approach can tolerate considerably larger fraction of mismatches than existing approaches, and enables estimation of the noise level as well as the fraction of mismatches. Inference for the resulting estimator (standard errors, confidence intervals) can be based on established theory for composite likelihood estimation. Along the way, we also propose a statistical test for the presence of mismatches and establish its consistency under suitable conditions.

Linear Regression without Correspondences via Concave Minimization

“…where u A,1 is the first column of U A and v 1 = ȳ ⊗ u A,1 . We can solve (13) via sorting in O(m log(m)) time [7]. Finally, it is also cheap to compute the smallest rectangle R • that contains the constraint set of ( 12), the latter being…”

“…We compute R • as follows. For i ∈ [n], the minimum and maximum of z i = v ⊤ i vec(B), say l • i and u • i , can be obtained in a similar way as in (13). This computation amounts to solving 2n sorting problems.…”

“…The next tractable instance is when the data are partially shuffled, as occurs, e.g., in record linkage [8]- [14]. In such a case, the estimation of x * may be performed via convex ℓ 1 robust regression [8] or a pseudo-likelihood approach [13], these tolerate at most 50% or 70% mismatches, respectively. When n ≥ 2 and the data are fully shuffled, as in point set registration [15], [16] or signal estimation using distributed sensors [17]- [19], ( 2) is strongly NP-hard [7], [20].…”

Permuted and Unlinked Monotone Regression in $\mathbb{R}^d$: an approach based on mixture modeling and optimal transport

“…In computer vision, a common task is to identify corresponding pairs of images, with one image arising as a distorted image of the other [5]; in this context, the function f * may represent a specific combination of distortions (e.g., scaling, rotations, blur, etc.). Specific instances of (1) that have received considerable attention lately are unlabeled sensing or linear regression with unknown permutation, e.g., [6,7,8,9,10,11,12,13,14,15,16] in which case f * is an affine transformation (albeit not necessarily from R d to R d ). Among these works, the papers [11,16] discuss applications in record linkage [17,18,19], specifically post-linkage data analysis [20,21,22].…”

“…Specific instances of (1) that have received considerable attention lately are unlabeled sensing or linear regression with unknown permutation, e.g., [6,7,8,9,10,11,12,13,14,15,16] in which case f * is an affine transformation (albeit not necessarily from R d to R d ). Among these works, the papers [11,16] discuss applications in record linkage [17,18,19], specifically post-linkage data analysis [20,21,22]. The papers [23,24] consider the case in which X n and Y n are points in the unit sphere in R d and f * is a unitary map with applications in automated translation between different word embeddings.…”