Parameter Optimization for Nadaraya-Watson Kernel Regression Method with Small Samples

Abstract

Abstract-Many current regression algorithms have unsatisfactory prediction accuracy with small samples. To solve this problem, a regression algorithm based on Nadaraya-Watson kernel regression (NWKR) is proposed. The proposed method advocates parameter selection directly from the standard deviation of training data, optimized with leave-one-out crossvalidation (LOO-CV). Good generalization performance of the proposed parameter selection is demonstrated empirically using small sample regression problems with Gaussian noise. The results show that proposed parameter optimization method is more robust and accurate than other methods for different noise levels and different sample sizes, and indicate the importance of Vapnik's ε-insensitive loss for regression problems with small samples.

Reduced-order modeling of supersonic fuel–air mixing in a multi-strut injection scramjet engine using machine learning techniques

“…For more complex datasets, e.g. datasets with strong variations in sample density, strategies such as the leave-one-out cross validation (LOO-CV) method [46,47] or Bayesian optimization can be used to estimate the optimal bandwidths for all objective functions.…”