Estimation of model accuracy in CASP13

Cheng, Jianlin; Choe, Myong‐Ho; Elofsson, Arne; Han, Kun‐Sop; Hou, Jie; Maghrabi, Ali H. A.; McGuffin, Liam J.; Menéndez-Hurtado, David; Olechnovič, Kliment; Schwede, Torsten; Studer, Gabriel; Uziela, Karolis; Venclovas, C̆eslovas; Wallner, Björn

doi:10.1002/prot.25767

Cited by 93 publications

(117 citation statements)

References 61 publications

Supporting

Mentioning

115

Contrasting

Unclassified

Order By: Relevance

“…We condense the information in a scatterplot with the AUCs obtained for the partial ROC plotted against the AUCs' partial PR analyses. Within the top performing regime of this performance domain “QMEANDisCo3”, “QMEANDisCo2,” “ModFOLD6,” and “ModFOLD7‐lDDT” are clearly in the lead (Figure C), followed by “QMEAN3,” “ModFOLD4,” and “VoroMQA.” All these methods clearly outperform the “Baseline Potential” based on a full‐atomic statistical potential of mean force as has been reported earlier (Table ). In CAMEO, the current threshold of lDDT has been selected based on earlier unpublished investigations and discussions with the CAMEO participants.…”

Section: Resultssupporting

confidence: 74%

“…A very important way to communicate a model's quality to the end user is by assigning correct per‐residue confidence estimates . “SWISS‐MODEL,” “IntFOLD3‐TS,” and “Robetta” are providing good to very good error estimates with “SWISS‐MODEL” and “IntFOLD3‐TS” peaking at 0.87 and 0.85 as analyzed by averaging per‐model ROC AUCs.…”

Section: Resultsmentioning

confidence: 99%

“…Historic development of quality estimation tools. The improvements are impressive spanning early developments and recent approaches over the last 22 years, from well‐known tools such as PROSA, Verify3D, DFIRE to the latest contestants such as ProQ3, ModFOLD7_lDDT and QMEANDisCo3 . The years are assigned roughly to the best server of a particular year.…”

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO)

et al. 2019

Self Cite

View full text Add to dashboard Cite

Critical blind assessment of structure prediction techniques is crucial for the scientific community to establish the state of the art, identify bottlenecks, and guide future developments. In Critical Assessment of Techniques in Structure Prediction (CASP), human experts assess the performance of participating methods in relation to the difficulty of the prediction task in a biennial experiment on approximately 100 targets. Yet, the development of automated computational modeling methods requires more frequent evaluation cycles and larger sets of data. The “Continuous Automated Model EvaluatiOn (CAMEO)” platform complements CASP by conducting fully automated blind prediction evaluations based on the weekly pre‐release of sequences of those structures, which are going to be published in the next release of the Protein Data Bank (PDB). Each week, CAMEO publishes benchmarking results for predictions corresponding to a set of about 20 targets collected during a 4‐day prediction window. CAMEO benchmarking data are generated consistently for all methods at the same point in time, enabling developers to cross‐validate their method's performance, and referring to their results in publications. Many successful participants of CASP have used CAMEO—either by directly benchmarking their methods within the system or by comparing their own performance to CAMEO reference data. CAMEO offers a variety of scores reflecting different aspects of structure modeling, for example, binding site accuracy, homo‐oligomer interface quality, or accuracy of local model confidence estimates. By introducing the "bestSingleTemplate" method based on structure superpositions as a reference for the accuracy of 3D modeling predictions, CAMEO facilitates objective comparison of techniques and fosters the development of advanced methods.

show abstract

Section: Resultssupporting

confidence: 74%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO)

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…It is very difficult to assess progress in EMA methods over CASPs objectively because of the differences in the prediction targets and changes in the TS server methods that generate models. According to three top EMA participants of CASP13 (developers of “MULTICOM,” “ProQ,” and “ModFOLD” series), their new versions of EMA methods showed better performance than their previous versions that participated in CASP12, when the previous versions were tested on the same CASP13 EMA targets …”

Section: Assessment Results and Discussionmentioning

confidence: 99%

“…Such features of the models resulted in spuriously high performance of a pure consensus method compared to single‐model methods. Single‐model EMA methods showed progresses when compared to the previous versions individually on the same CASP13 targets, but performance relative to a reference method did not improve. This may be again due to the changed nature of the models generated in CASP13 by the new TS methods because the EMA methods were trained on the previous CASP models and methods.…”

Section: Resultsmentioning

confidence: 99%

Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning

et al. 2019

View full text Add to dashboard Cite

Scoring model structure is an essential component of protein structure prediction that can affect the prediction accuracy tremendously. Users of protein structure prediction results also need to score models to select the best models for their application studies. In Critical Assessment of techniques for protein Structure Prediction (CASP), model accuracy estimation methods have been tested in a blind fashion by providing models submitted by the tertiary structure prediction servers for scoring. In CASP13, model accuracy estimation results were evaluated in terms of both global and local structure accuracy. Global structure accuracy estimation was evaluated by the quality of the models selected by the global structure scores and by the absolute estimates of the global scores. Residue‐wise, local structure accuracy estimations were evaluated by three different measures. A new measure introduced in CASP13 evaluates the ability to predict inaccurately modeled regions that may be improved by refinement. An intensive comparative analysis on CASP13 and the previous CASPs revealed that the tertiary structure models generated by the CASP13 servers show very distinct features. Higher consensus toward models of higher global accuracy appeared even for free modeling targets, and many models of high global accuracy were not well optimized at the atomic level. This is related to the new technology in CASP13, deep learning for tertiary contact prediction. The tertiary model structures generated by deep learning pose a new challenge for EMA (estimation of model accuracy) method developers. Model accuracy estimation itself is also an area where deep learning can potentially have an impact, although current EMA methods have not fully explored that direction.

show abstract

Using physical features of protein core packing to distinguish real proteins from decoys

et al. 2020

View full text Add to dashboard Cite

The ability to consistently distinguish real protein structures from computationally generated model decoys is not yet a solved problem. One route to distinguish real protein structures from decoys is to delineate the important physical features that specify a real protein. For example, it has long been appreciated that the hydrophobic cores of proteins contribute significantly to their stability. We used two sources to obtain datasets of decoys to compare with real protein structures: submissions to the biennial Critical Assessment of protein Structure Prediction competition, in which researchers attempt to predict the structure of a protein only knowing its amino acid sequence, and also decoys generated by 3DRobot, which have user‐specified global root‐mean‐squared deviations from experimentally determined structures. Our analysis revealed that both sets of decoys possess cores that do not recapitulate the key features that define real protein cores. In particular, the model structures appear more densely packed (because of energetically unfavorable atomic overlaps), contain too few residues in the core, and have improper distributions of hydrophobic residues throughout the structure. Based on these observations, we developed a feed‐forward neural network, which incorporates key physical features of protein cores, to predict how well a computational model recapitulates the real protein structure without knowledge of the structure of the target sequence. By identifying the important features of protein structure, our method is able to rank decoy structures with similar accuracy to that obtained by state‐of‐the‐art methods that incorporate many additional features. The small number of physical features makes our model interpretable, emphasizing the importance of protein packing and hydrophobicity in protein structure prediction.

show abstract

Estimation of model accuracy in CASP13

Cited by 93 publications

References 61 publications

Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO)

Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO)

Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning

Using physical features of protein core packing to distinguish real proteins from decoys

Contact Info

Product

Resources

About