Universal Robust Regression via Maximum Mean Discrepancy

Abstract

Many datasets are collected automatically, and are thus easily contaminated by outliers. In order to overcome this issue, there was recently a regain of interest in robust estimation methods. However, most of these methods are designed for a specific purpose, such as estimation of the mean, or linear regression. We propose estimators based on Maximum Mean Discrepancy (MMD) optimization as a universal framework for robust regression. We provide non-asymptotic error bounds, and show that our estimators are robust to Huber-type contamination. We discuss the optimization of the objective functions via (stochastic) gradient descent in classical examples such as linear, logistic or Poisson regression. These results are illustrated by a set of simulations.

Discrepancy-based Inference for Intractable Generative Models using Quasi-Monte Carlo

“…The dataset consists of m = 2 16 points, from which a minibatch of 2 10 points is sampled at random at every iteration. Depending on the considered experiment, either n samples are generated using MC or n, n 3 4 , n 2 3 or n 1 2 are simulated using RQMC at every iteration. The optimisation algorithm is run for 3,000 iterations for every setting.…”

“…As could be reasonably expected, the RQMC-based estimator with n points is significantly more expensive than a MC with n points, but it is also much more accurate in l 2 error. Similarly, the RQMC-based estimators with n 2 3 or n 1 2 are less accurate in l 2 error but usually cheaper than MC with n points. More interestingly, we see that the RQMC estimator with n 3 4 points is both cheaper and more accurate than the MC estimator with n points for the Wasserstein and Sinkhorn divergences.…”

“…The first discrepancy we will consider is the maximum mean discrepancy (MMD) [38], which compares embeddings of probability distributions into reproducing kernel Hilbert spaces, and can be straightforwardly computed through evaluations of a kernel. This was studied by [15,20,21,3] in the context of MDE, and by [47,28,45,73,13] for the case where G θ is a neural network in particular. It was also used by [55,64,49,42,12] in the context of ABC.…”

Estimation of copulas via Maximum Mean Discrepancy

“…See [13,19] for the two-sample test problem, e.g. As a tool for parametric estimation, even though it was implicitly used in specific examples in machine learning [15], MMD has been studied as a general method for inference only recently [2,4,10,11]. In the latter papers, it appeared that MMD criteria lead to consistent estimators that are unconditionally robust to model misspecification.…”

“…Z 2 = z 2 is N (sz 2 , 1 − s 2 ), we can easily calculate ψ(z) := E exp(−Z 2 1 /(γ 2 /2))|Z 2 = z . Indeed, setting τ 2 := {1/γ 2 + 1/(1 − s 2 )} −1 , we have…”