CASP prediction center infrastructure and evaluation measures in CASP10 and CASP ROLL

Abstract

The Protein Structure Prediction Center at the University of California, Davis, supports the CASP experiments by identifying prediction targets, accepting predictions, performing standard evaluations, assisting independent CASP assessors, presenting and archiving results, and facilitating information exchange relating to CASP and structure prediction in general. We provide an overview of the CASP infrastructure implemented at the Center, and summarize standard measures used for evaluating predictions in the latest round of CASP. Several components were introduced or significantly redesigned for CASP10, in particular an improved assessors' common web-workspace; a Sphere Grinder visualization tool for analyzing local accuracy of predictions; brand new blocks for evaluation contact prediction and contact-assisted structure prediction; expanded evaluation and visualization tools for tertiary structure, refinement and quality assessment. Technical aspects of conducting the CASP10 and CASP ROLL experiments and relevant statistics are also provided.

Compare local pocket and global protein structure models by small structure patterns

“…In the tenth Critical Assessment of Techniques for Protein Structure Prediction (CASP10) [12], each predicted protein structure model was optimally superimposed to the native protein structure model, and the quality of the predicted model was measured by the alignment score calculated from the superimposed models. Totally, 103 target protein structures are predicted by servers (i.e., without human interactions), and 26, 135 pairs of predicted and native protein structure models (with at least 20 corresponding residue pairs) are conducted.…”

“…For each pair of predicted and native models conducted from CASP10, the GDT-TS scores [32] are calculated using our method and state-of-the-art LGA method [31]. Previously, the GDT-TS score and the LGA method have been used by CASP competitions since 1998 [32,16,10,12]. In this study, the GDT-TS scores are normalized by the number of corresponding residue pairs between predicted and native models, and N is set to be the smallest meaningful value of 3.…”

“…Indeed, such cases do not even exist for consecutive subsets. Therefore, for accuracy sensitive tasks such as the predicted protein structure model evaluation of CASP competitions [12], remote subsets are more suitable for the task.…”

“…Formally, the problem is to find the optimal superposition that rotates and translates the target structure model to fit its native structure model according to a scoring function. This is one of the essential tasks in the Critical Assessment of Techniques for Protein Structure Prediction (CASP) competitions [12,6,7]. Indeed, the success of the predicted protein structure model assessment process depends mainly on finding the optimal superposition.…”

“…To evaluate the performance of PROSTA-super, we compared it with LGA [31] using 11, 669 pairs of predicted and native local protein pocket models and 26, 135 pairs of predicted and native global protein structure models from the CASP10 dataset [12]. We found that the GDT-TS scores [32] calculated by PROSTA-super were significantly higher than those calculated by LGA for the protein pocket models.…”