Soft Tissue Simulation Environment to Learn Manipulation Tasks in Autonomous Robotic Surgery

Tagliabue, Eleonora; Pore, Ameya; Dall’Alba, Diego; Magnabosco, Enrico; Piccinelli, Marco; Fiorini, Paolo

doi:10.1109/iros45743.2020.9341710

Cited by 49 publications

(38 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Methods presented in Sec. IV are trained in a simulation environment which is an exact replica of the phantom, where the deformation properties of the fat tissue have been optimized to mimic those of the real synthetic tissue used in the experiments [12]. All the simulation experiments, including DRL training and dVRK control, are executed on a workstation equipped with an AMD Ryzen 3700X processor and NVIDIA TitanX GPU.…”

Section: Methodsmentioning

confidence: 99%

“…The proposed environment does not support deformable objects, which is a major limitation when simulating the surgical scenario. Recently, Tagliabue et al proposed a virtual framework called UnityFlexML for simulating deformable tissues that is well suited to train DRL methods [12]. We use this simulation framework to develop our LfD approach.…”

Section: Related Workmentioning

confidence: 99%

“…1). Registration between the simulated and real environment is guaranteed following the same registration process described in [12]. The joint values are sent to UnityFlexML through UDP sockets and the desired configuration is reached with direct kinematics.…”

Section: B Learning From Demonstrations (Lfd) Setupmentioning

confidence: 99%

See 2 more Smart Citations

Learning from Demonstrations for Autonomous Soft-tissue Retraction

Pore¹,

Tagliabue²,

Piccinelli³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The current research focus in Robot-Assisted Minimally Invasive Surgery (RAMIS) is directed towards increasing the level of robot autonomy, to place surgeons in a supervisory position. Although Learning from Demonstrations (LfD) approaches are among the preferred ways for an autonomous surgical system to learn expert gestures, they require a high number of demonstrations and show poor generalization to the variable conditions of the surgical environment. In this work, we propose an LfD methodology based on Generative Adversarial Imitation Learning (GAIL) that is built on a Deep Reinforcement Learning (DRL) setting. GAIL combines generative adversarial networks to learn the distribution of expert trajectories with a DRL setting to ensure generalisation of trajectories providing human-like behaviour. We consider automation of tissue retraction, a common RAMIS task that involves soft tissues manipulation to expose a region of interest. In our proposed methodology, a small set of expert trajectories can be acquired through the da Vinci Research Kit (dVRK) and used to train the proposed LfD method inside a simulated environment. Results indicate that our methodology can accomplish the tissue retraction task with human-like behaviour while being more sample-efficient than the baseline DRL method. Towards the end, we show that the learnt policies can be successfully transferred to the real robotic platform and deployed for soft tissue retraction on a synthetic phantom.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning from Demonstrations for Autonomous Soft-tissue Retraction

Pore¹,

Tagliabue²,

Piccinelli³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This application category includes papers working on cutting and debridement [61], [87], [90], [92], [95], [97] as well as the retraction and dissection of tissues [101], [131], [132], [155], [257], [265] or blood suction [226]. Also included is tissue palpation for locating tumors or vessels and more general tissue manipulation, as in [13]- [15], [17], [82], [83], [86], [98], [99], [154], [164], [225], and [241], with experiments sometimes using just common fabric as a phantom for tissue [94], [100].…”

Section: Instrument Controlmentioning

confidence: 99%

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

D'Ettorre

Mariani

Stilli

et al. 2021

IEEE Robot. Automat. Mag.

View full text Add to dashboard Cite

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.R obotic-assisted surgery is now well established in clinical practice and has become the gold-standard clinical treatment option for several clinical indications. The field of robotic-assisted surgery is expected to grow substantially in the next decade, with a range of new robotic devices emerging to address unmet clinical needs across different specialties. A vibrant surgical robotics research community is pivotal for conceptualizing such new systems as well as for developing and training the engineers and scientists to translate them into practice. The da Vinci Research Kit (dVRK), an academic and industry collaborative effort to repurpose decommissioned da Vinci surgical systems [Intuitive Surgical Inc. (ISI), California, USA] as a research platform for surgical robotics research, has been a key initiative for addressing a barrier to entry for new research groups in surgical robotics. In this article, we present an extensive review of the publications that have been facilitated by the dVRK over the past decade. We classify research efforts into different categories and outline some of the major challenges and needs for the robotics community to maintain and build upon this initiative.

show abstract

“…In our experiment, we lift the tissue from each grasping point to the maximum feasible extent and we record the point cloud representing the current state of the surface at regular steps of 10 mm, while increasing the lifting. The point cloud is acquired by an Intel RealSense D435 RGB-D camera and is automatically registered to the virtual geometry thanks the initial system calibration, which is performed following the same process described in [20]. The displacement field relative to the visible phantom surface is retrieved by computing a dense point-topoint matching problem between the acquired point clouds and the undeformed phantom surface.…”

Section: B Real World Phantom Datamentioning

confidence: 99%

Data-Driven Intra-Operative Estimation of Anatomical Attachments for Autonomous Tissue Dissection

Tagliabue

Dall’Alba

Pfeiffer

et al. 2021

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

The execution of surgical tasks by an Autonomous Robotic System (ARS) requires an up-to-date model of the current surgical environment, which has to be deduced from measurements collected during task execution. In this work, we propose to automate tissue dissection tasks by introducing a convolutional neural network, called BA-Net, to predict the location of attachment points between adjacent tissues. BA-Net identifies the attachment areas from a single partial view of the deformed surface, without any a-priori knowledge about their location. The proposed method guarantees a very fast prediction time, which makes it ideal for intra-operative applications. Experimental validation is carried out on both simulated and real world phantom data of soft tissue manipulation performed with the da Vinci Research Kit (dVRK). The obtained results demonstrate that BA-Net provides robust predictions at varying geometric configurations, material properties, distributions of attachment points and grasping point locations. The estimation of attachment points provided by BA-Net improves the simulation of the anatomical environment where the system is acting, leading to a median simulation error below 5mm in all the tested conditions. BA-Net can thus further support an ARS by providing a more robust test bench for the robotic actions intra-operatively, in particular when replanning is needed. The method and collected dataset are available at https://gitlab.com/altairLab/banet.

show abstract

Soft Tissue Simulation Environment to Learn Manipulation Tasks in Autonomous Robotic Surgery

Cited by 49 publications

References 13 publications

Learning from Demonstrations for Autonomous Soft-tissue Retraction

Learning from Demonstrations for Autonomous Soft-tissue Retraction

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

Data-Driven Intra-Operative Estimation of Anatomical Attachments for Autonomous Tissue Dissection

Contact Info

Product

Resources

About