Deep learning with 4D spatio-temporal data representations for OCT-based force estimation

Gessert, Nils; Bengs, Marcel; Schlüter, Matthias; Schlaefer, Alexander

doi:10.1016/j.media.2020.101730

Cited by 20 publications

(12 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In future work, tissue identification could be used to improve the generalization ability of the proposed deep learning models. Also, force prediction could be performed by estimating future force values based on past values as shown in [11].…”

Section: Resultsmentioning

confidence: 99%

Generalization of spatio-temporal deep learning for vision-based force estimation

Behrendt

Gessert

Schlaefer

2020

Current Directions in Biomedical Engineering

Self Cite

View full text Add to dashboard Cite

Robot-assisted minimally-invasive surgery is increasingly used in clinical practice. Force feedback offers potential to develop haptic feedback for surgery systems. Forces can be estimated in a vision-based way by capturing deformation observed in 2D-image sequences with deep learning models. Variations in tissue appearance and mechanical properties likely influence force estimation methods’ generalization. In this work, we study the generalization capabilities of different spatial and spatio-temporal deep learning methods across different tissue samples. We acquire several data-sets using a clinical laparoscope and use both purely spatial and also spatio-temporal deep learning models. The results of this work show that generalization across different tissues is challenging. Nevertheless, we demonstrate that using spatio-temporal data instead of individual frames is valuable for force estimation. In particular, processing spatial and temporal data separately by a combination of a ResNet and GRU architecture shows promising results with a mean absolute error of 15.450 compared to 19.744 mN of a purely spatial CNN.

show abstract

Section: Resultsmentioning

confidence: 99%

Generalization of spatio-temporal deep learning for vision-based force estimation

Behrendt

Gessert

Schlaefer

2020

Current Directions in Biomedical Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, the sensor needs a large space for use in an MIS environment. Gessert et al [28] proposed 4D deep learning with streams of volumes for OCT-based force estimation. A temporal history improves force estimation performance and predicts short-term force.…”

Section: Vision-based Force Estimationmentioning

confidence: 99%

Vision-Based Suture Tensile Force Estimation in Robotic Surgery

Jung

Kwak

Lim

2020

Sensors

View full text Add to dashboard Cite

Compared to laparoscopy, robotics-assisted minimally invasive surgery has the problem of an absence of force feedback, which is important to prevent a breakage of the suture. To overcome this problem, surgeons infer the suture force from their proprioception and 2D image by comparing them to the training experience. Based on this idea, a deep-learning-based method using a single image and robot position to estimate the tensile force of the sutures without a force sensor is proposed. A neural network structure with a modified Inception Resnet-V2 and Long Short Term Memory (LSTM) networks is used to estimate the suture pulling force. The feasibility of proposed network is verified using the generated DB, recording the interaction under the condition of two different artificial skins and two different situations (in vivo and in vitro) at 13 viewing angles of the images by changing the tool positions collected from the master-slave robotic system. From the evaluation conducted to show the feasibility of the interaction force estimation, the proposed learning models successfully estimated the tensile force at 10 unseen viewing angles during training.

show abstract

“…Feasibility in mapping the OCT surface deformation to forces was demonstrated [7]. Also, learning force estimates from full OCT volumes with CNNs has been studied [8,9] where promising results were achieved on exvivo data.…”

Section: Problemmentioning

confidence: 99%

Force estimation from 4D OCT data in a human tumor xenograft mouse model

Neidhardt

Gessert

Gosau

et al. 2020

Current Directions in Biomedical Engineering

Self Cite

View full text Add to dashboard Cite

Minimally invasive robotic surgery offer benefits such as reduced physical trauma, faster recovery and lesser pain for the patient. For these procedures, visual and haptic feedback to the surgeon is crucial when operating surgical tools without line-of-sight with a robot. External force sensors are biased by friction at the tool shaft and thereby cannot estimate forces between tool tip and tissue. As an alternative, vision-based force estimation was proposed. Here, interaction forces are directly learned from deformation observed by an external imaging system. Recently, an approach based on optical coherence tomography and deep learning has shown promising results. However, most experiments are performed on ex-vivo tissue. In this work, we demonstrate that models trained on dead tissue do not perform well in in vivo data. We performed multiple experiments on a human tumor xenograft mouse model, both on in vivo, perfused tissue and dead tissue. We compared two deep learning models in different training scenarios. Training on perfused, in vivo data improved model performance by 24% for in vivo force estimation.

show abstract

Deep learning with 4D spatio-temporal data representations for OCT-based force estimation

Cited by 20 publications

References 19 publications

Generalization of spatio-temporal deep learning for vision-based force estimation

Generalization of spatio-temporal deep learning for vision-based force estimation

Vision-Based Suture Tensile Force Estimation in Robotic Surgery

Force estimation from 4D OCT data in a human tumor xenograft mouse model

Contact Info

Product

Resources

About