Local Dimensionality Reduction for Non-Parametric Regression

Abstract

Locally-weighted regression is a computationally-efficient technique for non-linear regression. However, for high-dimensional data, this technique becomes numerically brittle and computationally too expensive if many local models need to be maintained simultaneously. Thus, local linear dimensionality reduction combined with locally-weighted regression seems to be a promising solution. In this context, we review linear dimensionalityreduction methods, compare their performance on non-parametric locally-linear regression, and discuss their ability to extend to incremental learning. The considered methods belong to the following three groups: (1) reducing dimensionality only on the input data, (2) modeling the joint input-output data distribution, and (3) optimizing the correlation between projection directions and output data. Group 1 contains principal component regression (PCR); group 2 contains principal component analysis (PCA) in joint input and output space, factor analysis, and probabilistic PCA; and group 3 contains reduced rank regression (RRR) and partial least squares (PLS) regression. Among the tested methods, only group 3 managed to achieve robust performance even for a non-optimal number of components (factors or projection directions). In contrast, group 1 and 2 failed for fewer components since these methods rely on the correct estimate of the true intrinsic dimensionality. In group 3, PLS is the only method for which a computationally-efficient incremental implementation exists.

Dielectric spectroscopy‐based estimation ofVCDinCHOcell culture

“…In LWPLS, estimation of unknown samples is made by constructing a local PLS model for each sample based on a subset of their nearest neighbors in a calibration set . In theory, this can provide better regression performance for a dataset with a mixed or nonlinear distribution . In our case LWPLS was tested to model the heterogeneous correlation between dielectric spectra and the VCD at different growth phases.…”

Principal Component Analysis on Spatial Data: An Overview

“…LWPCA and GWPCA: Moving from Global PCA to Local PCA in Attribute or Geographic Space LWPCA (Tipping and Bishop 1999;Skočaj, Leonardis, and Bischof 2007;Hoffmann, Schaal, and Vijayakumar 2009;Charlton et al 2010) is applied to the situation when the data are not described well by a universal set of PCs but where there are localized regions in attribute data space where a suitably localized set of PCs provide a better description. That is, in different parts of the data space, a different set of PCs is needed.…”

Model learning for robot control: a survey

“…The application of dimensionality reduction is based on the insight that the useful information in the data often lies on a low-dimensional manifold of the original input space. Dimensionality reduction methods have proven to be a powerful method for model learning in high-dimensional robot systems (Hoffman et al 2009;Schaal et al 2002).…”