A real-time compression library for microscopy images

Abstract

Fluorescence imaging techniques such as single molecule localization microscopy, highcontent screening and light-sheet microscopy are producing ever-larger datasets, which poses increasing challenges in data handling and data sharing. Here, we introduce a realtime compression library that allows for very fast (beyond 1 GB/s) compression and decompression of microscopy datasets during acquisition. In addition to an efficient lossless mode, our algorithm also includes a lossy option, which limits pixel deviations to the intrinsic noise level of the image and yields compression ratio of up to 100-fold. We present a detailed performance analysis of the different compression modes for various biological samples and imaging modalities. opened new perspectives in biology by increasing the speed of imaging, the number of specimens or the resolution of the observed structures. Even though these methods bring undeniable advantages, the data production speed and experiment sizes (Fig. 1a, Supplementary Table 1) are increasing in such a fast pace that in many cases data handling quickly becomes a bottleneck for new discoveries [12]-[14]. A straightforward solution to this problem is to perform image compression. Nonetheless, this typically implies incompatibilities with certain software packages, slow compression speed, and only moderate file size reduction for lossless methods. Although the compression ratio (original size / compressed size) can be substantially increased with lossy compression algorithms, their use is often discouraged [15] as the degree of information loss heavily depends on the image content and cannot be explicitly controlled.To address these challenges, we developed a new compression library called B 3 D, which is capable of extremely fast compression and decompression of large microscopy datasets. Our library is built on the CUDA architecture [16] for GPU-based compression, which not only enables high processing speed, but also relieves load on the central processing unit, allowing compression directly during image acquisition. The algorithm has two main components. First, a prediction is made for each pixel based on the neighboring pixel values, and second, the prediction errors are run-length and Huffman encoded to effectively reduce the data size (Supplementary Note and Supplementary Fig. 1). We compared our algorithm's performance with TIFF (LZW), JPEG2000, and the speed-optimized KLB [17] by measuring compression speed, decompression speed and resulting file size (Fig. 1b). Only B³D is capable of handling the sustained high data rate of modern sCMOS cameras typically . CC-BY 4.0 International license not peer-reviewed) is the author/funder. It is made available under a

Tissue clearing and its applications in neuroscience

“…The reliance on ImgLib2, which is designed to write efficient image analysis algorithms in Java independent of the data type (8-bit, 16-bit, RGB and so on), dimensionality (1D to nD) and storage strategy (memory, local or remote file systems), makes it possible to deploy through BigDataViewer complex image processing pipelines on virtually limitless data. The B 3 D format offers a complementary approach to high-performance data compression using graphics processing unit-based compute unified device architecture (CUDA) processing 127 . Although B 3 D does not use a block-based architecture, it offers excellent write/read speeds on the order of 1,000 MB per second and includes lossless and lossy compression schemes.…”

A guidebook for DISCO tissue clearing

“…We based our data storage on the Lempel‐Ziv‐Welch (LZW) lossless compression, which to our experience is the most compatible across the various image analyzing and rendering software. Fortunately, a growing number of advanced scientific‐grade data storage options (to store metadata besides plain imagery) became available such as the Hierarchical Data Formats (e.g., HDF5), the KLB format (Amat et al , 2015), the B 3 D format (preprint: Balázs et al , 2017) which will hopefully eventually be implemented in all widely used research software.…”

“…The advantage of these stitchers is that after the initial setup, the computer automatically finds the displacement offset pairs and finally reconstructs the entire 3D scan optionally in multiple resolution levels. Fortunately, both alternatives are capable of parallel graphics‐processing‐unit‐based acceleration of displacement finding (preprint: Balázs et al , 2017) and automatic compression of the reconstructed volumes, which significantly speeds up the entire process. BigStitcher is available as a Fiji plugin and offers additional functionalities besides stitching (including multi‐angle stitching, deconvolution, and illumination selection) to compensate for optical effects.…”

Rapid pathology of lumpectomy margins with open-top light-sheet (OTLS) microscopy

“…For example, a two-channel OTLS dataset of a 2-cm by 2-cm specimen is ~81.6 GB large. We are exploring various methods [90] to compress the OTLS datasets without causing significant image degradation. Finally, our current multi-step data acquisition and processing pipeline is not well-suited for routine clinical practice and a simpler consolidated software package with improved efficiency is necessary.…”