Modal Regression via Direct Log-Density Derivative Estimation

Abstract

Modal regression is aimed at estimating the global mode (i.e., global maximum) of the conditional density function of the output variable given input variables, and has led to regression methods robust against heavy-tailed or skewed noises. The conditional mode is often estimated through maximization of the modal regression risk (MRR). In order to apply a gradient method for the maximization, the fundamental challenge is accurate approximation of the gradient of MRR, not MRR itself. To overcome this challenge, in this paper, we take a novel approach of directly approximating the gradient of MRR. To approximate the gradient, we develop kernelized and neural-network-based versions of the least-squares log-density derivative estimator, which directly approximates the derivative of the log-density without density estimation. With direct approximation of the MRR gradient, we first propose a modal regression method with kernels, and derive a new parameter update rule based on a fixed-point method. Then, the derived update rule is theoretically proved to have a monotonic hill-climbing property towards the conditional mode. Furthermore, we indicate that our approach of directly approximating the gradient is compatible with recent sophisticated stochastic gradient methods (e.g., Adam), and then propose another modal regression method based on neural networks. Finally, the superior performance of the proposed methods is demonstrated on various artificial and benchmark datasets.

A Quick Maximum Power Point Tracking Method Using an Embedded Learning Algorithm for Photovoltaics on Roads

“…As mentioned in a previous study [14], the KDE used in modal regression should approximate the number of peak points of the distribution, rather than the distribution itself. Let…”

Properties of Mean Shift

“…Various variants of the MS algorithm have been proposed. The CMS algorithm can be regarded as a weighted version of the MS algorithm having the weights w i related to the independent variable part of the sample points x i (see [11], [12], [13] for details). Therefore, the CMS algorithm is included in the general MS algorithm (5) as a special case.…”

“…The conditional mean shift (CMS) algorithm [11], [12], [13], a variant of the MS algorithm, is a representative technique for nonparametric modal regression, which has been applied to analysis of traffic data [12], [13] and weather data [14]. The CMS algorithm can be regarded as a weighted version of the conventional MS algorithm with the weights determined by the values of the independent variables in the samples, and it estimates modes of a conditional PDF of the dependent variables conditioned on the independent variables.…”

Modal regression using kernel density estimation: A review

“…Plug‐ins from a KDE have been applied in estimations of many structures (Chen, ; Scott, ) such as the regression function(Nadaraya, ; Watson, ), modes (Chacón & Duong, ; Chen, Genovese, & Wasserman, ), ridges (Chen et al, ; Genovese, Perone‐Pacifico, Verdinelli, & Wasserman, ), and level sets (Chen, Genovese, & Wasserman, ; Rinaldo & Wasserman, ). An alternative way of estimating the multimodal regression was proposed in Sasaki, Ono, and Sugiyama ().…”