Quantification of biases in predictions of protein stability changes upon mutations

Abstract

Bioinformatics tools that predict protein stability changes upon point mutations have made a lot of progress in the last decades and have become accurate and fast enough to make computational mutagenesis experiments feasible, even on a proteome scale. Despite these achievements, they still suffer from important issues that must be solved to allow further improving their performances and utilizing them to deepen our insights into protein folding and stability mechanisms. One of these problems is their bias towards the learning datasets which, being dominated by destabilizing mutations, causes predictions to be better for destabilizing than for stabilizing mutations. We thoroughly analyzed the biases in the prediction of folding free energy changes upon point mutations (∆∆G 0 ) and proposed some unbiased solutions. We started by constructing a dataset S sym of experimentally measured ∆∆G 0 s with an equal number of stabilizing and destabilizing mutations, by collecting mutations for which the structure of both the wild type and mutant protein is available. On this balanced dataset, we assessed the performances of fifteen widely used ∆∆G 0 predictors. After the astonishing observation that almost all these methods are strongly biased towards destabilizing mutations, especially those that use blackbox machine learning, we proposed an elegant way to solve the bias issue by imposing physical symmetries under inverse mutations on the model structure, which we implemented in PoPMuSiC sym . This new predictor constitutes an efficient trade-off between accuracy and absence of biases. Some final considerations and suggestions for further improvement of the predictors are discussed.

Assessment of methods for predicting the effects of PTEN and TPMT protein variants

“…We assessed the ability of computational methods to predict the effects of individual nonsynonymous variants on the VSP assayderived stability of two proteins. Our assessment differs from previous assessments (Khan & Vihinen, 2010;Potapov et al, 2009;Pucci et al, 2018;Thiltgen & Goldstein, 2012) in two ways. First, the use of high-throughput VSP assay data as the gold standard enabled evaluations free from ascertainment bias; that is, mutations were not explicitly targeted to protein positions of interest.…”

“…A key advantage of VSP assays is that they are applicable to a wide range of proteins and can measure effects of all possible mutations at all positions. They particularly help overcome issues of overrepresentation of to-alanine (Magliery, 2015) and destabilizing mutations (Pucci, Bernaerts, Kwasigroch, & Rooman, 2018), the underrepresentation of "inverse" mutations (Thiltgen & Goldstein, 2012), and general errors in curation (Yang et al, 2018). Thus, VSP assays are an attractive alternative to existing data sources for the development and validation of computational methods.…”

Predicting changes in protein thermodynamic stability upon point mutation with deep 3D convolutional neural networks

“…In contrast, FoldX predicted that 83.2% of pathogenic variants are destabilizing. As already demonstrated in previous studies, the bias is likely because FoldX was parameterized on an experimental ∆∆G data set dominated by destabilizing mutations (Pucci et al, 2018;Thiltgen and Goldstein, 2012;Usmanova et al, 2018).…”

“…We systematically compared ThermoNet with fifteen ∆∆G predictors on a common, balanced data set S sym to evaluate their performance and degree of bias with respect to the ∆∆G symmetry between direct and reverse mutations (Methods). The S sym data set, which was constructed previously for assessing the biases of ∆∆G predictors (Pucci et al, 2018), consists of experimentally measured ∆∆G values for 342 direct mutations from fifteen protein chains for which the structures of both the wild-type and mutant proteins have been resolved by X-ray crystallography with a resolution of 2.5 Å or better. To increase the size of this data set and evaluate prediction bias of ThermoNet, we generated a reverse mutation for each of the direct mutations, thus totaling 684 mutations.…”

“…For example, physics-based methods are computationally demanding and low-throughput; these challenges have largely prevented them from being applied to large-scale protein engineering and variant interpretation tasks. In addition, several studies have highlighted significant bias in the predictions of machine learning-based methods; they tend to predict mutations as destabilizing more often than stabilizing (Fariselli et al, 2015;Pucci et al, 2018;Thiltgen and Goldstein, 2012;Usmanova et al, 2018). The main source of this bias likely comes from the facts that the training sets are dominated by experimentderived destabilizing mutations and that machine learning methods are prone to overfitting to training sets (Fang, 2019;Pucci et al, 2018).…”