Assessing the accuracy of contact predictions in CASP13

Abstract

The accuracy of sequence‐based tertiary contact predictions was assessed in a blind prediction experiment at the CASP13 meeting. After 4 years of significant improvements in prediction accuracy, another dramatic advance has taken place since CASP12 was held 2 years ago. The precision of predicting the top L/5 contacts in the free modeling category, where L is the corresponding length of the protein in residues, has exceeded 70%. As a comparison, the best‐performing group at CASP12 with a 47% precision would have finished below the top 1/3 of the CASP13 groups. Extensively trained deep neural network approaches dominate the top performing algorithms, which appear to efficiently integrate information on coevolving residues and interacting fragments or possibly utilize memories of sequence similarities and sometimes can deliver accurate results even in the absence of virtually any target specific evolutionary information. If the current performance is evaluated by F‐score on L contacts, it stands around 24% right now, which, despite the tremendous impact and advance in improving its utility for structure modeling, also suggests that there is much room left for further improvement.

A fully open-source framework for deep learning protein real-valued distances

“…Previous studies have shown that long-range contacts, i.e. pairs separated by at least 23 residues in the sequence, are the most informative pairs for accurate reconstruction [24,25]. Hence, we designed our evaluation metrics focusing on long-range distances (see Figure 2).…”

“…In a recent work [26] the Cheng group have reported that selecting or evaluating top L/x contacts does not work well for all proteins. Also, in the most recent CASP competition, the accessors of the contact prediction category have discussed many reasons for considering to evaluate top NC contacts instead of fewer contacts [25]. Although the P N C metric is not discussed for most contact prediction method papers, we believe that as more and more accurate contact prediction methods are being developed, this will emerge as a more informative and reliable metric.…”

“…closer/smaller distances are more important). The Fisher's group refer to such interactions as 'interaction hubs' [25]. It may immediately follow that the commonly used machine learning loss functions such as mean squared error or mean absolute error are unfit for this purpose because they focus on optimizing the longer distance values before the shorter ones.…”

“…Hence, we limit our evaluation to these 9 domains. Consistent with the practice, we predict distances and contacts for the full target and evaluate on the domains [24,25]. For all our predictions, we use the alignments predicted by the trRosetta method, as our input.…”

Predicting changes in protein thermodynamic stability upon point mutation with deep 3D convolutional neural networks

“…Convolutional neural networks are a type of deep-learning model commonly used in computer vision applications. They have recently proven to perform well in residue contact prediction (Kandathil et al, 2019a;Schaarschmidt et al, 2018;Shrestha et al, 2019;Wang et al, 2018) and protein tertiary structure prediction (Greener et al, 2019;Kandathil et al, 2019b;Xu, 2019;Xu and Wang, 2019). We selected CNNs for protein structure-based ∆∆G prediction because we formulated this problem as a computer vision problem by treating protein structures as if they were 3D images.…”