Evaluating the predictions of the protein stability change upon single amino acid substitutions for the FXN CAGI5 challenge

Abstract

Frataxin (FXN) is a highly conserved protein found in prokaryotes and eukaryotes that is required for efficient regulation of cellular iron homeostasis. Experimental evidence associates amino acid substitutions of the FXN to Friedreich Ataxia, a neurodegenerative disorder. Recently, new thermodynamic experiments have been performed to study the impact of somatic variations identified in cancer tissues on protein stability. The Critical Assessment of Genome Interpretation (CAGI) data provider at the University of Rome measured the unfolding free energy of a set of variants (FXN challenge data set) with far‐UV circular dichroism and intrinsic fluorescence spectra. These values have been used to calculate the change in unfolding free energy between the variant and wild‐type proteins at zero concentration of denaturant (ΔΔGnormalH2O). The FXN challenge data set, composed of eight amino acid substitutions, was used to evaluate the performance of the current computational methods for predicting the ΔΔitalicGnormalH2O value associated with the variants and to classify them as destabilizing and not destabilizing. For the fifth edition of CAGI, six independent research groups from Asia, Australia, Europe, and North America submitted 12 sets of predictions from different approaches. In this paper, we report the results of our assessment and discuss the limitations of the tested algorithms.

Limitations and challenges in protein stability prediction upon genome variations: towards future applications in precision medicine

“…The prediction of protein stability changes upon variation is essential for protein design and precision medicine. The current methods, although far from being perfect, achieved sufficient levels of performance to complement experimental studies [68] . However, several issues need to be addressed to enhance such levels of performance, in particular: increasing the quality and the size of the current datasets, adding more carefully curated experimental data; building methods that are intrinsically anti-symmetric (ΔΔG(A->B) = -ΔΔG(B->A)); for machine-learning approaches, it is essential that the model training has to be performed using low levels of sequence identity between learning and testing sets.…”

“…These measurements were used to calculate the change in unfolding free energy between the variant and wild-type proteins at zero concentration of denaturant (ΔΔ G H2O ). The experimental dataset [67] , including eight amino acid substitutions, was used to evaluate the performance of the web-available tools for predicting the value of ΔΔ G H2O associated with the variants and to classify them as destabilizing or not-destabilizing [68] . Eight performance measures were applied to test the methods, five for assessing the predictions in regression (Pearson and Spearman Correlation Coefficients, Kendall-Tau Coefficient, Root Mean Square and Mean Absolute Errors) and three for the classification performance, i.e.…”

Reports from the fifth edition of CAGI: The Critical Assessment of Genome Interpretation

“…Over all CAGI editions, the plurality of challenges have been on the interpretation of isolated missense variants, and CAGI5 continues that trend. There are assessment, data provider, and participant papers for the prediction of the destabilizing effect of missense mutations in a cancer‐relevant protein (Frataxin, with biophysical measurements of protein stability; Petrosino et al, ; Savojardo, Petrosino et al, ; Strokach, Corbi‐Verge, & Kim, ); on the effect of missense changes in a human calmodulin, assayed using a high‐throughput yeast complementation assay (Zhang et al, ); the effect of missense mutations related to schizophrenia in human Pericentriolar Material 1 ( PCM1 ), using a zebrafish development model (Miller, Wang, & Bromberg, ; Monzon et al, ); the effect of missense mutations in two cancer‐related proteins, PTEN and TPMT , on intracellular protein levels, measured in a high‐throughput assay (Pejaver et al, ); and the effect of missense changes in a monogenic disease related protein, acid alpha‐glucosidase ( GAA ), with measurements of total intracellular enzyme activity (Adhikari, ). Three participant papers describe results on all the missense challenges (Garg & Pal, ; Katsonis & Lichtarge, ; Savojardo, Babbi et al, ).…”

Are machine learning based methods suited to address complex biological problems? Lessons from CAGI‐5 challenges

“…This variant is associated to a very low experimental ΔΔG value of −9.5 kcal/mol. As stated during the official assessment, the protein variant p.Trp173Cys corresponds to a clear unfolded state of the protein as experimentally determined Savojardo et al, 2019). For this reason, the data providers assigned These values are much lower than the ones reported in Table 1 and closer to performances reported by INPS-3D in other benchmarks (Savojardo et al, 2016).…”

“…This variant is associated to a very low experimental ΔΔ G value of −9.5 kcal/mol. As stated during the official assessment, the protein variant p.Trp173Cys corresponds to a clear unfolded state of the protein as experimentally determined (Petrosino et al, ; Savojardo et al, ). For this reason, the data providers assigned to the variant an arbitrary Δ G of 0 kcal/mol and, as a consequence, a very low ΔΔ G value.…”