The impact of data reduction and lossy image formats on electron microscope images

Abstract

Recent advances in instrumentation and software for cryoEM have increased the applicability and utility of this method. Coupled with the adoption of automated pipelines, significant infrastructure support is required to sustain high throughput workflows. In particular, data generation rates may outpace the ability to deploy data storage and archival solutions. We have investigated what effects data compression and conversion to different file formats have on the ability to obtain high resolution cryoEM reconstructions. Standard lossless data compression strategies have a high impact on reducing the size of direct detector electron counting movie stacks, but provide more modest gains for aligned summed images. We show that EM images can be compressed using standard lossy methods to reduce file storage to 5-10% of the size of the original aligned sum or movie stack file and yet still retain enough information such that modern image processing pipelines will provide sub-2Å reconstructions from the compressed data.

Electron-event representation data enable efficient cryoEM file storage with full preservation of spatial and temporal resolution

“…The standard output from Falcon cameras includes both gain correction and real-space anti-aliasing and consequently these files do not compress efficiently. A current approach to image handling from other cameras is to store LZW-TIFFcompressed four-bit super-resolution images, applying the gain reference and performing Fourier cropping after decompression (Eng et al, 2019). This approach reduces the file sizes for exposure fractions substantially compared with the uncompressed exposure fractions shown in Fig.…”

“…However, the real-time output of EER data from the camera avoids saving extremely large uncompressed intermediate files even temporarily, which would make workflows prohibitively complicated. Lossy compression approaches have also been shown to reduce file sizes when the complete preservation of information is not required (Eng et al, 2019). Consequently, conventional files that are smaller than the EER format can be produced, but doing so requires sacrificing temporal or spatial resolution.…”

Mimotope discovery as a tool to design a vaccine against Zika and dengue viruses

“…Raw movies were frame-aligned and dose-weighted using Motion-Cor. The resulting images were compressed into jpeg images to save space (Eng et al, 2019). Compressed jpgs were restored to mrc format for further processing in cryoSPARC (Punjani et al, 2017).…”