On the Upper Bounds of the Real-Valued Predictions

Abstract

Predictions are fundamental in science as they allow to test and falsify theories. Predictions are ubiquitous in bioinformatics and also help when no first principles are available. Predictions can be distinguished between classifications (when we associate a label to a given input) or regression (when a real value is assigned). Different scores are used to assess the performance of regression predictors; the most widely adopted include the mean square error, the Pearson correlation (ρ), and the coefficient of determination (or [Formula: see text]). The common conception related to the last 2 indices is that the theoretical upper bound is 1; however, their upper bounds depend both on the experimental uncertainty and the distribution of target variables. A narrow distribution of the target variable may induce a low upper bound. The knowledge of the theoretical upper bounds also has 2 practical applications: (1) comparing different predictors tested on different data sets may lead to wrong ranking and (2) performances higher than the theoretical upper bounds indicate overtraining and improper usage of the learning data sets. Here, we derive the upper bound for the coefficient of determination showing that it is lower than that of the square of the Pearson correlation. We provide analytical equations for both indices that can be used to evaluate the upper bound of the predictions when the experimental uncertainty and the target distribution are available. Our considerations are general and applicable to all regression predictors.

Limitations and challenges in protein stability prediction upon genome variations: towards future applications in precision medicine

“…Some studies provided a theoretical estimation of the upper bound of the Pearson correlation as a function of the average uncertainty of the data (σ) and the standard deviation of the dataset ( see Fig. 1 C) [61] , [62] . As an example, the popular datasets S2648 and VariBench have a σ DB < 2 kcal/mol, leading to an upper bound for the Pearson correlation coefficient of ~ 0.8 and a lower bound for the root mean square error between experimental and predicted ΔΔ G values of ~ 1 kcal/mol.…”

Performance of Regression Models as a Function of Experiment Noise

“…Since , the upper bound for becomes , and we refer to this upper bound as hereafter. Despite claims that no ML model could perform better than this upper bound, 15 , 16 by comparing the equations for and , it is clear that estimates are lower than both estimates as well as achieved ML model performance.…”

“…Methods to estimate this upper bound are underdeveloped, although some progress has been made recently. 15 , 16 Moreover, the resources invested into model development have diminishing returns on model performance as one approaches the upper bound. Knowing the best expected MSE or R 2 (i.e.…”

“…Data shown were randomly generated, se i denotes standard error of sample i. R LG 2 is the expected upper bound for the coefficient of determination R 2 derived in this study (see section "Estimating the theoretical upper bound of regression model performance").this assumption, the expectation of the upper bound for MSE is 2 2 σ y and R 2 is σ is the average variance of all sample noise and σ DB 2 is the variance of the real values (without noise). Since σ σ σ claims that no ML model could perform better than this upper bound,15,16 by comparing the equations for R Monte Carlo simulations. Briefly, a random dataset x y i , obs,i{ }was generated from a known real function f x ( ) with added experimental noise σ y i , (see section "Monte Carlo simulations on expected R 2 score").…”

A Deep-Learning Sequence-Based Method to Predict Protein Stability Changes Upon Genetic Variations

“…Regarding the results presented in Figures 3 and 4, it is worth noticing that the maximum achievable Pearson's correlation is not necessarily equal to 1, as usually thought. It may be far lower depending on the experimental uncertainty and the ∆∆G distributions [36,37]. In particular, when considering the different experiments on the same variants included in the Protherm database or in manually-cleaned datasets, the expected Pearson upper bound is in the range of 0.70-0.85 [36].…”