Methods of model accuracy estimation can help selecting the best models from decoy sets: Assessment of model accuracy estimations in CASP11

Abstract

The paper presents assessment of the model accuracy estimation methods participating in CASP11. The results of the assessment are expected to be useful to both – developers of the methods and users who way too often are presented with structural models without annotations of accuracy. The main emphasis is placed on the ability of techniques to identify the best models from among several available. Bivariate descriptive statistics and ROC analysis are used to additionally assess the overall correctness of the predicted model accuracy scores, the correlation between the predicted and observed accuracy of models, the effectiveness in distinguishing between good and bad models, the ability to discriminate between reliable and unreliable regions in models, and the accuracy of the coordinate error self-estimates. A rigid-body measure (GDT_TS) and three local-structure based scores (LDDT, CADaa, and Sphere Grinder) are used as reference measures for evaluating methods' performance. Consensus methods, taking advantage of the availability of several models for the same target protein, perform well on the majority of tasks. Methods that predict accuracy on the basis of a single model perform comparably to consensus methods in picking the best models and in the estimation of how accurate is the local structure. More groups than in previous experiments submitted reasonable error estimates of their own models, most likely in response to a recommendation from CASP and the increasing demand from users.

The SWISS-MODEL Repository—new features and functionality

“…Several approaches for model quality estimation have been developed in recent years to assist in determining the expected quality of models (see e.g. (31) for a recent benchmark of various tools).…”

Evaluation system and web infrastructure for the second cryo-EM model challenge

“…Clustering methods require model sets with many (preferably diverse) models and assume that agreement of models (or regions of models) may be an indication of correctness. In general, methods of this class perform well and are more accurate than single-model methods (Kryshtafovych et al, 2016b). However, if models are too diverse (i.e., clusters of similar models are too loose) or only a very few models are correct (i.e., the most popular clusters are dominated by poor models), these methods may generate wrong results, and the single-model methods may be a better choice.…”

On protein structure, function and modularity from an evolutionary perspective

“…[8] Smaller elements can always be partitioned out of a whole, but if it is done without any underlying assumptions one risks ending up with bits and pieces which cannot be attributed any real meaning. An example is the partitioning of a protein sequence into its amino acids; we know the chemical aspects of most amino acids but the procedure does not give us much insight into the function of the protein by studying the amino acids separately.…”

“…Transformed BL21(DE3) were grown at 37 C until OD600 of 2.0, induced by 1 mM isopropyl b-D-1-thiogalactopyranoside, harvested, and lysed by sonication in lysis buffer (300 mM sodium chloride, 20 mM Tris [pH 8.0], 20 mM imidazole, 1 mM tris(2-carboxyethyl) phosphine (TCEP), 10% glycerol, 1 mg/ml lysozyme, 5 U/ml DNase I, and 13 Protease Inhibitor tablets (Roche Diagnostics), and purified on Ni-nitrilotriacetic acid beads at 4 C. Where required for SAXS, the 6x-His tag was cleaved on the column. The flow-through was purified on a Superdex-75 column (GE Healthcare) equilibrated in lysis buffer and analyzed by MALDI-TOF and SDS-PAGE.…”

“…account structural similarity of models, that is, consensus-based methods. 8 Since CASP11, however, they perform on par or even better than the consensus methods in some of the tasks. 8 The first successful attempt of model quality estimation, in the context of CASP, was when the first meta-predictor, CAFASP-CON-SENSUS, was introduced in CASP4.…”