Structured illumination microscopy (SIM) surpasses the optical diffraction limit and offers a two-fold enhancement in resolution over diffraction limited microscopy. However, it requires both intense illumination and multiple acquisitions to produce a single high-resolution image. Using deep learning to augment SIM, we obtain a five-fold reduction in the number of raw images required for super-resolution SIM, and generate images under extreme low light conditions (at least 100× fewer photons). We validate the performance of deep neural networks on different cellular structures and achieve multi-color, live-cell super-resolution imaging with greatly reduced photobleaching.

Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparoscopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected, the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.

Increasing multi-site infant neuroimaging datasets are facilitating the research on understanding early brain development with larger sample size and bigger statistical power. However, a joint analysis of cortical properties (e.g., cortical thickness) is unavoidably facing the problem of nonbiological variance introduced by differences in MRI scanners. To address this issue, in this paper, we propose cycle-consistent adversarial networks based on spherical cortical surface to harmonize cortical thickness maps between different scanners. We combine the spherical U-Net and CycleGAN to construct a surface-to-surface CycleGAN (S2SGAN). Specifically, we model the harmonization from scanner X to scanner Y as a surface-to-surface translation task. The first goal of harmonization is to learn a mapping G X : X → Y such that the distribution of surface thickness maps from G X (X) is indistinguishable from Y. Since this mapping is highly under-constrained, with the second goal of harmonization to preserve individual differences, we utilize the inverse mapping G Y : Y → X and the cycle consistency loss to enforce G Y (G X (X)) ≈ X (and vice versa). Furthermore, we incorporate the correlation coefficient loss to guarantee the structure consistency between the original and the generated surface thickness maps. Quantitative evaluation on both synthesized and real infant cortical data demonstrates the superior ability of our method in removing unwanted scanner effects and preserving individual differences simultaneously, compared to the state-of-the-art methods.

Convolutional Neural Networks (CNNs) have been providing the state-of-the-art performance for learning-related problems involving 2D/3D images in Euclidean space. However, unlike in the Euclidean space, the shapes of many structures in medical imaging have a spherical topology in a manifold space, e.g., brain cortical or subcortical surfaces represented by triangular meshes, with large inter-subject and intrasubject variations in vertex number and local connectivity. Hence, there is no consistent neighborhood definition and thus no straightforward convolution/transposed convolution operations for cortical/subcortical surface data. In this paper, by leveraging the regular and consistent geometric structure of the resampled cortical surface mapped onto the spherical space, we propose a novel convolution filter analogous to the standard convolution on the image grid. Accordingly, we develop corresponding operations for convolution, pooling, and transposed convolution for spherical surface data and thus construct spherical CNNs. Specifically, we propose the Spherical U-Net architecture by replacing all operations in the standard U-Net with their spherical operation counterparts. We then apply the Spherical U-Net to two challenging and neuroscientifically important tasks in infant brains: cortical surface parcellation and cortical attribute map development prediction. Both applications demonstrate the competitive performance in the accuracy, computational efficiency, and effectiveness of our proposed Spherical U-Net, in comparison with the state-of-the-art methods.