A Statistical Learning Approach to Modal Regression

Abstract

This paper studies the nonparametric modal regression problem systematically from a statistical learning view. Originally motivated by pursuing a theoretical understanding of the maximum correntropy criterion based regression (MCCR), our study reveals that MCCR with a tending-to-zero scale parameter is essentially modal regression. We show that nonparametric modal regression problem can be approached via the classical empirical risk minimization. Some efforts are then made to develop a framework for analyzing and implementing modal regression. For instance, the modal regression function is described, the modal regression risk is defined explicitly and its Bayes rule is characterized; for the sake of computational tractability, the surrogate modal regression risk, which is termed as the generalization risk in our study, is introduced. On the theoretical side, the excess modal regression risk, the excess generalization risk, the function estimation error, and the relations among the above three quantities are studied rigorously. It turns out that under mild conditions, function estimation consistency and convergence may be pursued in modal regression as in vanilla regression protocols such as mean regression, median regression, and quantile regression. However, it outperforms these regression models in terms of robustness as shown in our study from a re-descending M-estimation view. This coincides with and in return explains the merits of MCCR on robustness. On the practical side, the implementation issues of modal regression including the computational algorithm and the tuning parameters selection are discussed. Numerical assessments on modal regression are also conducted to verify our findings empirically.

Modal Regression via Direct Log-Density Derivative Estimation

“…To make our approach clearer, we adopt the terminologies in Feng et al [2017]. Let us assume that the output variable y is generated from the following model:…”

“…Recently, modal regression has been gathering a great deal of attention due to the clear advantages over conventional regression methods based on the conditional mean [Sager and Thisted, 1982, Collomb et al, 1986, Carreira-Perpiñán, 2000, Einbeck and Tutz, 2006, Yao et al, 2012, Chen et al, 2016, Feng et al, 2017, Wang et al, 2017. Modal regression can be roughly divided into unimodal and multimodal regression.…”

“…Modal regression can be roughly divided into unimodal and multimodal regression. The goal of the unimodal regression is to estimate the global mode (i.e., global maximum) of the conditional density, leading to regression methods robust against skewed or heavy-tailed noises [Sager and Thisted, 1982, Collomb et al, 1986, Yao et al, 2012, Feng et al, 2017, while the conventional conditional mean estimation could be vulnerable to these nonGaussian noises. On the other hand, multimodal regression is aimed at estimating local modes (i.e., local maxima) of the conditional density, and simultaneously finds multiple functional relationships between input and output variables which the conditional mean cannot capture [Carreira-Perpiñán, 2000, Einbeck and Tutz, 2006, Chen et al, 2016, Sasaki et al, 2016.…”

“…The mode of the conditional density has been often estimated through maximization of the empirical modal regression risk (MRR), which is defined as the sample average of the conditional density (or the joint density) [Sager and Thisted, 1982, Yao et al, 2012, Feng et al, 2017. A naive approach in modal regression takes a two-step approach of firstly approximating the empirical MRR via conditional (or joint) density estimation (e.g., by kernel density estimation), and secondly of maximizing the approximated risk by some gradient method.…”

“…Another approach in modal regression employs a surrogate risk of MRR [Lee, 1989, Yao and Li, 2014, Feng et al, 2017, Wang et al, 2017. The advantage of this approach is that high-dimensional density estimation can be avoided.…”