Learning with correntropy-induced losses for regression with mixture of symmetric stable noise

Abstract

In recent years, correntropy and its applications in machine learning have been drawing continuous attention owing to its merits in dealing with non-Gaussian noise and outliers. However, theoretical understanding of correntropy, especially in the learning theory context, is still limited. In this study, we investigate correntropy based regression in the presence of non-Gaussian noise or outliers within the statistical learning framework. Motivated by the practical way of generating non-Gaussian noise or outliers, we introduce mixture of symmetric stable noise, which include Gaussian noise, Cauchy noise, and their mixture as special cases, to model non-Gaussian noise or outliers. We demonstrate that under the mixture of symmetric stable noise assumption, correntropy based regression can learn the conditional mean function or the conditional median function well without resorting to the finite-variance or even the finite first-order moment condition on the noise. In particular, for the above two cases, we establish asymptotic optimal learning rates for correntropy based regression estimators that are asymptotically of type O(n −1 ). These results justify the effectiveness of the correntropy based regression estimators in dealing with outliers as well as non-Gaussian noise. We believe that the present study makes a step forward towards understanding correntropy based regression from a statistical learning viewpoint, and may also shed some light on robust statistical learning for regression. h ′ with h(t) = −σ 2 exp(−t 2 /σ 2 ), t ∈ R, and the second inequality is due to Theorem 1. On the other hand, for any g 1

New Insights Into Learning With Correntropy-Based Regression

“…As a result, imposing stronger moment conditions may not help improve the established convergence rates of the estimator, implying the existence of an inherent bias in mean regression. These novel insights can help cement the theoretical correntropy framework developed recently in Feng et al (2015Feng & Ying, 2020;Feng & Wu, 2020).…”

“…Therefore, under the zero median assumption, f is in fact the conditional median function as the conditional mean function may not even be defined. According to Feng and Ying (2020), in this case, MCCR can learn the conditional median function f well in the sense that Ep ε|X (Y − f z,σ (X )) → Ep ε|X (Y − f (X )) implies f z,σ → f with a proper fixed σ . Moreover, fast exponential-type convergence rates can be established.…”

“…For instance, inspired by the work in Hu, Fan, Wu, and Zhou (2013) and Fan, Hu, Wu, and Zhou (2016), Feng et al (2015) demonstrated that MCCR can deal with mean regression robustly in the sense that only a fourth-moment condition on the response variable is needed to guarantee its convergence. Such a moment condition is further relaxed to the (1 + )th order moment condition in Feng and Wu (2020), encompassing the case when the noise possesses infinite variance; Feng, Fan, and Suykens (2020) showed that MCCR performs modal regression under certain restrictions to the noise; using Huber's contamination model for modeling outliers, Feng and Ying (2020) make some efforts to explain the outlier robustness of MCCR and shows that it can be used to learn f in the presence of outliers.…”

Maximum Correntropy Criterion with Distributed Method

“…First, our distributed result provides the optimal rates by requiring a large robust parameter σ. In practice, a moderate σ may be enough to ensure a good learning performance in robust estimation as shown by [17]. It is therefore of interest to investigate the convergence properties of distributed version of algorithm (5) when σ is chosen as a constant or σ(N) → 0 as N approaches ∞.…”

“…The commonly used robust losses mainly include adaptive Huber loss [11], gain function [12], minimum error entropy [13], exponential squared loss [14], etc. Among them, the Maximum Correntropy Criterion (MCC) is widely employed as an efficient alternative to the ordinary least squares method which is suboptimal in the non-Gaussian and non-linear signal processing situations [15][16][17][18][19]. Recently, MCC has been studied extensively in the literature and is widely adopted for many learning tasks, e.g., wind power forecasting [20] and pattern recognition [19].…”