Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Abstract

This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

Improving cryo-EM structure validation

“…alidation metrics are an essential part of how biomolecular structures are vetted before publication and interpreted after publication, but judging the accuracy of features within structural models is almost always challenging. Lawson and colleagues report on a community-wide model building and validation challenge, highlight progress in developing robust validation of atomic models of biological macromolecules, and offer recommendations on how to improve cryogenic electron microscopy (cryo-EM) structure validation 1 .…”

“…The search for a complete set of robust validation metrics for cryo-EM maps and models is far from over, but a recent model validation challenge, reported in this issue 1 , marks substantial progress. In the challenge, four high-quality 3D maps were distributed and set as targets for model building.…”

“…Nevertheless, a "resolution revolution" seeded by Lukosz in the 1960s and heralded by Hell, Gustafsson and Betzig in the 2000s introduced "super-resolution" (SR) to the scientific vernacular. In this issue of Nature Methods, Qiao et al 1 continue the revolution by riding the tidal wave of deep learning (DL) -a framework rooted in the 1940s 2 that has only recently enjoyed the computational muscle required by any but the simplest tasks 3and present alchemic results: transforming low-resolution, low-contrast, noisy images into super-resolved, high contrast, clean micrographs.…”

“…Qiao et al 's achievement is threefold. First, they collect an exceptional training dataset 1 , an invaluable public resource for new method development, consisting of matched noisy, low-resolution images and high-quality, super-resolved, structured illumination microscopy (SIM, a variant of SR microscopy) reconstructions. Second, they introduce two DL architectures, termed deep Fourier channel attention networks (DFCAN) and deep Fourier generative adversarial networks (DFGAN), which, as their names imply, learn feature representations in the Fourier domain; and finally, they apply the networks to both the perennial problem of SIM reconstruction from nine low-quality images and the more fantastical concept of single-image SR (SISR) 4 , in which an SR image is inferred entirely from a single diffraction-limited, or lower resolution, image.…”

Cryo-EM Map–Based Model Validation Using the False Discovery Rate Approach

“…The agreement is either quantified as Manders' overlap coefficient in SMOC (Joseph et al, 2016), real space cross-correlation coefficient in PHENIX local CCC (Afonine et al, 2018), or a score of atomic resolvability in MapQ (Pintilie 2020). One of the observations from the recent model challenge is that the absolute values of some of these metrics are sensitive to the map resolution (Lawson et al, 2021). One reason is the underlying sensitivity of the metric toward differences in the shape of map distributions at different resolutions.…”