Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks

Abstract

Purpose: In surgical navigation, pre-operative organ models are presented to surgeons during the intervention to help them in efficiently finding their target. In the case of soft tissue, these models need to be deformed and adapted to the current situation by using intra-operative sensor data. A promising method to realize this are real-time capable biomechanical models.Methods: We train a fully convolutional neural network to estimate a displacement field of all points inside an organ when given only the displacement of a part of the organ's surface. The network trains on entirely synthetic data of random organ-like meshes, which allows us to generate much more data than is otherwise available. The input and output data is discretized into a regular grid, allowing us to fully utilize the capabilities of convolutional operators and to train and infer in a highly parallelized manner. Results: The system is evaluated on in-silico liver models, phantom liver data and human in-vivo breathing data. We test the performance with varying material parameters, organ shapes and amount of visible surface. Even though the network is only trained on synthetic data, it adapts well to the various cases and gives a good estimation of the internal organ displacement. The inference runs at over 50 frames per second.Conclusions: We present a novel method for training a data-driven, real-time capable deformation model. The accuracy is comparable to other registration methods, it adapts very well to previously unseen organs and does not need to be re-trained for every patient. The high inferring speed makes this method useful for many applications such as surgical navigation and real-time simulation.

A case study: impact of target surface mesh size and mesh quality on volume-to-surface registration performance in hepatic soft tissue navigation

“…We speculate that this also depends on the deformation and its extent, and possibly the organ of interest. Previous work that employs a different spatial discretization, particularly a voxel representation, has found that a target surface of 20% of the visible organ surface was necessary and corresponded to the largest drop in error [ 15 ]. Indeed, this corresponds to the size of the unfiltered stitched surface (Endo) used in [ 23 ] and is roughly half the amount of accessible liver surface that is accessible.…”

“…However, the liver may deform under pressure exerted by surgical instruments, characterizing soft tissue deformation. Characteristic hepatic tissue deformations occur either in the first operative steps (upon mobilization of the liver or transection of the liver parenchyma), or during laparoscopy by abnormally high pressure of the capnoperitoneum (12)(13)(14)(15). The grade of deformation depends on the liver stiffness and the patient-specific liver structure.…”

“…However, the quality of the geometry (e.g., the data noise) and how to improve it has not yet been addressed. A couple of works addressed the factor of visible surface in decreasing increments for two different spatial discretizations: a triangular mesh [16] and a voxel grid [15]. Yet, the impact of noise and the geometry quality on registration results remains open.…”

“…Yet, it is until recently that the first reference data were made available via the Stereo Correspondence and Reconstruction of Endoscopic Data Sub-Challenge Part of the Endoscopic Vision Challenge 2019 https://endovissub2019-scared.grand-challenge.org. Moreover, only one related work addressed the question of the required visible surface to obtain good registration results [15]. However, this work addressed this factor in relation to another spatial discretization of the geometry, i.e., voxel grid.…”

A Learnt Approach for the Design of Magnetically Actuated Shape Forming Soft Tentacle Robots

“…Utilizing FEM to simulate the interaction between magnetic and mechanical forces has been successfully demonstrated in previous work [21] and been successfully applied to CMs [8]. In addition, the use of ANNs as surrogates of FEMs has also been reported; for example in [26] and [27] convolutional neural networks are used to fully recreate FE simulations of a liver and an arterial wall respectively. These works exhibit results evidencing the effectiveness of deep learning in this particular sphere of research.…”