Robotic Surgery With Lean Reinforcement Learning

Abstract

As surgical robots become more common, automating away some of the burden of complex direct human operation becomes ever more feasible. Model-free reinforcement learning (RL) is a promising direction toward generalizable automated surgical performance, but progress has been slowed by the lack of efficient and realistic learning environments. In this paper, we describe adding reinforcement learning support to the da Vinci Skill Simulator, a training simulation used around the world to allow surgeons to learn and rehearse technical skills. We successfully teach an RL-based agent to perform sub-tasks in the simulator environment, using either image or state data. As far as we know, this is the first time an RL-based agent is taught from visual data in a surgical robotics environment. Additionally, we tackle the sample inefficiency of RL using a simple-to-implement system which we term hybrid-batch learning (HBL), effectively adding a second, long-term replay buffer to the Q-learning process. Additionally, this allows us to bootstrap learning from images from the data collected using the easier task of learning from state. We show that HBL decreases our learning times significantly.

LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

“…Across environments, adding depth information usually resulted in only minor increases in success rate of around 5% to 10%. These results are aligned with the findings of Barnoy et al (2021) that compare various different image-based observation types on a reach and a suturing task in RALS. The learning curves for these experiments on the spatial reasoning track are shown in Figure 12.…”

Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot

“…Surgical robots nowadays assist surgeons to conduct minimally invasive interventions in clinical routine [1]. Automating surgical tasks has been increasingly desired to improve surgical efficiency, with many efforts being made on different tasks such as suturing [2][3][4], endoscope control [5][6][7], tissue manipulation [8][9][10] and pattern cutting [11][12][13]. Recently, reinforcement learning (RL) approaches have exhibited high scalability to learn diverse control policies and yielded promising performance in surgical automation [14][15][16][17][18][19][20], but typically require extensive data collection to solve a task if no prior knowledge is given.…”