R2 Metric Dynamics for K-Nearest Neighbors Regression Model Trained on Series of Different Sizes

Abstract

An R2 score or a coefficient of determination is used often as a metric to evaluate regression models. It can be applied solely but usually it is combined with other metrics in order to increase accuracy of a model evaluation. The goal of the work is to research the dynamics of the R2 score of a K-Nearest Neighbors regression model trained on series of different sizes in order to propose a new approach to increase the robustness and accuracy of the model evaluation when the R2 score metric is used solely. Typically, a value of the R2 score metric above 0.8 is considered to be sufficient while an evaluated model is considered to be accurate enough. However, such a way of R2 score interpretation to may lead to model’s accuracy misevaluation, which is shown in the proposed paper. The results obtained clearly display that R2 score can vary significantly in some cases depending on the samples selected to test part of a series used for model evaluation. The mentioned variation can contribute to model’s accuracy overestimation, which, in turn can lead to incorrect results of model application. The known methods to make model estimation more accurate involve use of other metrics. Instead, this paper focuses on increase of model’s accuracy estimation without the necessity of using other metrics. The R2 score dynamics is examined using 25000 cycles of the K-Nearest Neighbors regression model training and evaluation. Selection of samples to a training or test part of a series has been done randomly. For all the experiments quantity of neighbors is fixed and equals to the default value of n_neighbors=5 of the KNeighborsRegressor method provided by the Sklearn library. The paper both states and proves a hypothesis that the R2 score variation is expected to increase with series size reduction and the variation is supposed to be observed for models trained on the same series because of training/test samples selection randomness. The experiments carried out allowed to propose an alternative approach that did not require any supplementary metrics. The proposed approach considers application of the R2 score along with its variation that must not exceed 0.2 for the K-Nearest Neighbors regression model.