Estimation of Interaction Forces in Robotic Surgery using a Semi-Supervised Deep Neural Network Model

Abstract

Providing force feedback as a feature in current Robot-Assisted Minimally Invasive Surgery systems still remains a challenge. In recent years, Vision-Based Force Sensing (VBFS) has emerged as a promising approach to address this problem. Existing methods have been developed in a Supervised Learning (SL) setting. Nonetheless, most of the video sequences related to robotic surgery are not provided with ground-truth force data, which can be easily acquired in a controlled environment. A powerful approach to process unlabeled video sequences and find a compact representation for each video frame relies on using an Unsupervised Learning (UL) method. Afterward, a model trained in an SL setting can take advantage of the available ground-truth force data. In the present work, UL and SL techniques are used to investigate a model in a Semi-Supervised Learning (SSL) framework, consisting of an encoder network and a Long-Short Term Memory (LSTM) network. First, a Convolutional Auto-Encoder (CAE) is trained to learn a compact representation for each RGB frame in a video sequence. To facilitate the reconstruction of high and low frequencies found in images, this CAE is optimized using an adversarial framework and a L1-loss, respectively. Thereafter, the encoder network of the CAE is serially connected with an LSTM network and trained jointly to minimize the difference between ground-truth and estimated force data. Datasets addressing the force estimation task are scarce. Therefore, the experiments have been validated in a custom dataset. The results suggest that the proposed approach is promising.

A Review on Deep Learning in Minimally Invasive Surgery

“…They achieved a root-mean square error of 0.02N. Advancing in this work, Marban et al [100] proposed a semi-supervised learning approach consisting of an encoder network serially connected with an LSTM network. First, unlabelled video sequences are used to train the encoder network to extract visual features from the images.…”

Discussion on the possibility of multi-layer intelligent technologies to achieve the best recover of musculoskeletal injuries: Smart materials, variable structures, and intelligent therapeutic planning

“…Deep reinforcement learning policies were used in Pattern Cutting task to induce tension in the material as cutting proceeds (Thananjeyan et al, 2017). Force sensors were developed to estimate interaction forces in robotic surgery (Kesner and Howe, 2011;Marban et al, 2018). And a Semi-Supervised Deep Neural Network Model was applied to understand the operation pattern (Marban et al, 2018).…”

“…Force sensors were developed to estimate interaction forces in robotic surgery (Kesner and Howe, 2011;Marban et al, 2018). And a Semi-Supervised Deep Neural Network Model was applied to understand the operation pattern (Marban et al, 2018).…”

“…Force sensors in robotic surgery (Kesner and Howe, 2011) (D). Semi-Supervised Deep Neural Network Model (Marban et al, 2018).…”

CathSim: An Open-source Simulator for Autonomous Cannulation

“…Recent developments in machine learning are promising a greater degree of autonomy in many robotic systems. Such systems leverage deep learning architectures such as convolutional neural networks [28], [29], long-shortterm memory [30]- [32], and generative adversarial imitation learning [16] to facilitate force estimation and catheter segmentation. Whilst those systems confer a lower level autonomy through the use of robotic assistive features, the higher level autonomy is left unaddressed.…”