Transform- and multi-domain deep learning for single-frame rapid autofocusing in whole slide imaging

Abstract

A whole slide imaging (WSI) system has recently been approved for primary diagnostic use in the US. The image quality and system throughput of WSI is largely determined by the autofocusing process. Traditional approaches acquire multiple images along the optical axis and maximize a figure of merit for autofocusing. Here we explore the use of deep convolution neural networks (CNNs) to predict the focal position of the acquired image without axial scanning. We investigate the autofocusing performance with three illumination settings: incoherent Kohler illumination, partially coherent illumination with two plane waves, and one-plane-wave illumination. We acquire ~130,000 images with different defocus distances as the training data set. Different defocus distances lead to different spatial features of the captured images. However, solely relying on the spatial information leads to a relatively bad performance of the autofocusing process. It is better to extract defocus features from transform domains of the acquired image. For incoherent illumination, the Fourier cutoff frequency is directly related to the defocus distance. Similarly, autocorrelation peaks are directly related to the defocus distance for two-plane-wave illumination. In our implementation, we use the spatial image, the Fourier spectrum, the autocorrelation of the spatial image, and combinations thereof as the inputs for the CNNs. We show that the information from the transform domains can improve the performance and robustness of the autofocusing process. The resulting focusing error is ~0.5 µm, which is within the 0.8-µm depth-of-field range. The reported approach requires little hardware modification for conventional WSI systems and the images can be captured on the fly without focus map surveying. It may find applications in WSI and time-lapse microscopy. The transform- and multi-domain approaches may also provide new insights for developing microscopy-related deep-learning networks. We have made our training and testing data set (~12 GB) open-source for the broad research community.

Autofocusing technologies for whole slide imaging and automated microscopy

“…Deep learning has been demonstrated as a powerful tool for solving inverse problems. With the advent of accelerated computing and deep learning frameworks such as TensorFlow and PyTorch, researchers have also explored various deep learning-based solutions for autofocusing [21,92,[139][140][141][142][143][144][145][146][147][148][149][150]. As shown in Figure 11, the reported deep-learning solutions can be, in general, categorized into two groups.…”

“…The first group is to predict the defocus distance or to locate the out‐of‐focus regions based on one or more input defocused images [21, 92–94, 140, 141, 144, 146, 147, 149, 150]. For example, Jiang et al employed a convolutional neural network (CNN) to estimate the defocus distance based on the transform‐ and multi‐domain inputs [141]. By adding the Fourier spectrum and the autocorrelation of the spatial image as the input, the performance and the robustness can be improved compared to that only with the spatial image as the input.…”

Whole slide imaging system using deep learning-based automated focusing

“…Jiang et al [15] explore the use of CNNs for the purpose of microscope auto focusing. Given a sample image anywhere in the focus stack, the trained CNN should be able to estimate the optimal distance to be moved vertically (either up or down) to reach the optimal focus position.…”

“…The test data consists of two different types of samples, prepared with different staining protocols at different sites. It was observed in [15] that the trained CNN has relatively poor performance on the test set, when trained with only the RGB images, especially on the test set with different protocol of preparation. To counter this, they propose the usage of spectral domain and multi domain inputs.…”

Solving Fourier ptychographic imaging problems via neural network modeling and TensorFlow

“…Different gradientdescent-based algorithms can be used in the backward pass, including momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, and Adam [32]. The use of neural networks for tackling microscopy problems is a rapidly growing research field with various applications [33][34][35][36][37].…”