On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy

Abstract

Cryo-electron microscopy (cryoEM) has become a well established technique to elucidate the 3D structures of biological macromolecules. Projection images from thousands of macromolecules that are assumed to be structurally identical are combined into a single 3D map representing the Coulomb potential of the macromolecule under study. This article discusses possible caveats along the image-processing path and how to avoid them to obtain a reliable 3D structure. Some of these problems are very well known in the community. These may be referred to as sample-related (such as specimen denaturation at interfaces or non-uniform projection geometry leading to underrepresented projection directions). The rest are related to the algorithms used. While some have been discussed in depth in the literature, such as the use of an incorrect initial volume, others have received much less attention. However, they are fundamental in any data-analysis approach. Chiefly among them, instabilities in estimating many of the key parameters that are required for a correct 3D reconstruction that occur all along the processing workflow are referred to, which may significantly affect the reliability of the whole process. In the field, the term overfitting has been coined to refer to some particular kinds of artifacts. It is argued that overfitting is a statistical bias in key parameter-estimation steps in the 3D reconstruction process, including intrinsic algorithmic bias. It is also shown that common tools (Fourier shell correlation) and strategies (gold standard) that are normally used to detect or prevent overfitting do not fully protect against it. Alternatively, it is proposed that detecting the bias that leads to overfitting is much easier when addressed at the level of parameter estimation, rather than detecting it once the particle images have been combined into a 3D map. Comparing the results from multiple algorithms (or at least, independent executions of the same algorithm) can detect parameter bias. These multiple executions could then be averaged to give a lower variance estimate of the underlying parameters.

Image processing tools for the validation of CryoEM maps

“…However, it is misled by systematic errors appearing in both halves. 8 Despite the plethora of alternative validation methods, these are seldom used due to the difficulty of accessing them conveniently and smoothly. Scipion 38 is an appropriate platform for this evaluation as it integrates all the methods described in this paper, a total of 37, and provides effective ways of allowing them to interoperate.…”

Emerging Themes in CryoEM─Single Particle Analysis Image Processing

“…If this is the case, the resulting map will not represent the structure we try to solve, and any interpretation of its biological informational content will be wrong. This situation is known in the field as overfitting, and in ref 168 we show that this is caused by bias in the estimation of the various parameters involved in the image processing.…”

“…If there are only two and they disagree, the most we can do is discard this input image as we cannot be sure which of the two is the correct estimate of the underlying model. The interested reader may consult ref for an expanded discussion of bias and variance in the estimation of parameters in CryoEM and how what we normally call overfitting is caused by bias in the estimation of the parameters.…”

“…Although methods such as the gold standard (splitting randomly the set of particles in two subsets with half the images each, which are then independently processed to reconstruct two maps) are used to detect overfitting, they do not guarantee the absence of systematic errors committed in both data halves. 168 Validation methods may analyze different features of the map itself, or for studies that propose an atomic model, they may consider the map and the structural model simultaneously. We should comment that early work on this topic included a report of a specially convened task force (the Validation Task Force), including a set of good practices to reduce the number of incorrect structures, 61 and a review of some validation tools can be seen in ref 133.…”

“…However, other regions in the micrograph may spuriously correlate with the kind of object we look for, resulting in false positives. A data set with a significant amount of false positives is rather dangerous because of the so-called Einstein-from-noise effect , (the average of noise particles aligned to a reference looks like the reference) and represents a threat in the SPA workflow due to the possibility to produce wrong but very reproducible structures as a consequence of the results being biased . A similar effect is observed if an incorrect macromolecular template is used for picking the particles.…”