Systematic Artifacts in Support Vector Regression-Based Compound Potency Prediction Revealed by Statistical and Activity Landscape Analysis

Abstract

Support vector machines are a popular machine learning method for many classification tasks in biology and chemistry. In addition, the support vector regression (SVR) variant is widely used for numerical property predictions. In chemoinformatics and pharmaceutical research, SVR has become the probably most popular approach for modeling of non-linear structure-activity relationships (SARs) and predicting compound potency values. Herein, we have systematically generated and analyzed SVR prediction models for a variety of compound data sets with different SAR characteristics. Although these SVR models were accurate on the basis of global prediction statistics and not prone to overfitting, they were found to consistently mispredict highly potent compounds. Hence, in regions of local SAR discontinuity, SVR prediction models displayed clear limitations. Compared to observed activity landscapes of compound data sets, landscapes generated on the basis of SVR potency predictions were partly flattened and activity cliff information was lost. Taken together, these findings have implications for practical SVR applications. In particular, prospective SVR-based potency predictions should be considered with caution because artificially low predictions are very likely for highly potent candidate compounds, the most important prediction targets.

Correcting Systematic Underprediction of Biochemical Oxygen Demand in Support Vector Regression

“…With respect to studies that reported systematic underprediction affecting peak values (e.g., Balfer and Bajorath 2015;Garsole and Rajurkar 2015;Granata et al 2016), this study was unable to access their data sets to determine their data distributions and verify the causes of underprediction. Nevertheless, the problem of underprediction affecting extreme-high dependent values is present in the modeling results of these studies, which corroborate the observation here that only underprediction is present in the prediction of extreme-high values.…”

“…In the case of the 2000-2014 data set for Yuen Long Creek, extreme-high BODs are subject to underprediction. Inherently, the systematic underprediction results from an SVR attempting to minimize both prediction errors and model complexity (Balfer and Bajorath 2015). As extreme-high values often represent only a small proportion of the data set, the SVR will algorithmically tolerate prediction errors in an effort to derive a sufficiently complex model that provides accurate predictions for the majority of the data-that is, the nonextreme values.…”

“…For instance, Garsole and Rajurkar (2015) and Granata et al (2016) identified that the SVR tends to underpredict peak hydrological discharges. A pharmaceutical modeling study by Balfer and Bajorath (2015) noted that the SVR underpredicted the potency value of highly potent compounds. One approach to resolve systematic underprediction is to increase the proportion of extreme-high values in the data set prior to building the model.…”

Data-driven design of B20 alloys with targeted magnetic properties guided by machine learning and density functional theory

“…It is important to recognize that extrapolation is considered a challenging problem for ML methods, especially when these methods are used without an adaptive learning scheme or iterative feedback loop [43]. Balfer and Bajorath [44] showed that when SVR methods are used for building quantitative structure-activity relationships, these methods systematically underestimate the true value of high-potency compounds due to the regularization term in the loss function that balances the trade-off between model complexity and its ability to generalize to an unseen data point. Therefore, our ML prediction for Snsubstituted FeGe needs further attention and cannot be discarded solely on the basis of ML predictions.…”

Automated support vector regression

“…The earliest work with QSPR was the prediction of efficacy relationship models . Some recent work and representative work with QSPR are prediction of activity cliffs, ligand efficiencies, compound potency, the regulative role of atomic autocorrelated electronegativities and polarizabilities in beta 2 potency, lipophilicity, and detonation velocity . Balabin et al compared SVR with artificial neural network regression using NIR data obtained from 14 sets of petroleum fuels and products .…”