Improving the visualization of electron-microscopy data through optical flow interpolation

Abstract

Technical developments in neurobiology have reached a point where the acquisition of high resolution images representing individual neurons and synapses becomes possible. For this, the brain tissue samples are sliced using a diamond knife and imaged with electron-microscopy (EM). However, the technique achieves a low resolution in the cutting direction, due to limitations of the mechanical process, making a direct visualization of a dataset difficult. We aim to increase the depth resolution of the volume by adding new image slices interpolated from the existing ones, without requiring modifications to the EM image-capturing method. As classical interpolation methods do not provide satisfactory results on this type of data, the current paper proposes a re-framing of the problem in terms of motion volumes, considering the depth axis as a temporal axis. An optical flow method is adapted to estimate the motion vectors of pixels in the EM images, and this information is used to compute and insert multiple new images at certain depths in the volume. We evaluate the visualization results in comparison with interpolation methods currently used on EM data, transforming the highly anisotropic original dataset into a dataset with a larger depth resolution. The interpolation based on optical flow better reveals neurite structures with realistic undistorted shapes, and helps to easier map neuronal connections.

Capillary networks and follicular marginal zones in human spleens. Three-dimensional models based on immunostained serial sections

“…It utilises dense optical flow [ 35 ]. A sparse interpolation for electron microscopy has been suggested before [ 36 ]. Recently, a VR visualisation tool was published for medical applications [ 37 ], but we used our own custom-written VR visualisation software.…”

The tempest in a cubic millimeter: Image-based refinements necessitate the reconstruction of 3D microvasculature from a large series of damaged alternately-stained histological sections

“…It appears to be viable to "heal" a missing section with existing mechanisms for the interpolation between sections [22,67]. Indeed, we also used optical flow for such interpolations.…”

“…Generation of intermediate sections for 3D reconstruction from microscopic data has been suggested before [22,67], however the method presented here differs in goal, details of blending, and in using registration prior to the computation of optical flow. A comparison with existing methods is in Section IV 1 b.…”

Three Dimensional Reconstruction

“…It utilises dense optical flow [Farnebäck, 2003]. A sparse interpolation for electron microscopy has been suggested before [Carata et al, 2013]. Recently, a VR visualisation tool was published for medical applications [Egger et al, 2017], but we used our own custom-written VR visualisation software (Chapter 5).…”

“…Ehrhardt et al [2007] use the method of Barron et al [1994] for optical flow computation, we utilise the method of Farnebäck [2003]. A sparse interpolation for electron microscopy has been suggested before [Carata et al, 2013]. Unlike our method it uses a sparse optical flow.…”