A Natural Language Interface for an Autonomous Camera Control System on the da Vinci Surgical Robot

Abstract

Positioning a camera during laparoscopic and robotic procedures is challenging and essential for successful operations. During surgery, if the camera view is not optimal, surgery becomes more complex and potentially error-prone. To address this need, we have developed a voice interface to an autonomous camera system that can trigger behavioral changes and be more of a partner to the surgeon. Similarly to a human operator, the camera can take cues from the surgeon to help create optimized surgical camera views. It has the advantage of nominal behavior that is helpful in most general cases and has a natural language interface that makes it dynamically customizable and on-demand. It permits the control of a camera with a higher level of abstraction. This paper shows the implementation details and usability of a voice-activated autonomous camera system. A voice activation test on a limited set of practiced key phrases was performed using both online and offline voice recognition systems. The results show an on-average greater than 94% recognition accuracy for the online system and 86% accuracy for the offline system. However, the response time of the online system was greater than 1.5 s, whereas the local system was 0.6 s. This work is a step towards cooperative surgical robots that will effectively partner with human operators to enable more robust surgeries. A video link of the system in operation is provided in this paper.

Evaluation of a Voice-Enabled Autonomous Camera Control System for the da Vinci Surgical Robot

“…A natural language (keyword-based) interface was added to the autocamera algorithm to provide more control of the camera movement to the user [26]. The Vosk speech recognition API, based on the Kaldi toolkit, was used to integrate the interface [28].…”

“…To enable seamless adjustments to autonomous camera behavior, we previously introduced a system that incorporates a voice assistant, allowing users to modify the base behavior of the autocamera system using voice commands, such as tracking the left or right tool with a specific voice command [26]. The current paper assesses the performance of the previously developed VACC system in comparison to the HOCC system within the context of our user study.…”

An Overview of Minimally Invasive Surgery Robots from the Perspective of Human–Computer Interaction Design

“…(1) The optimization of operation procedures and experience through the development of voice modules: Tang [55] established a voice-based HCI mode that enables robots to perform corresponding actions through voice control, solving the problem of poor co-ordination between assisting surgeons and primary surgeons. Maysara et al [56] developed a voice interface to assist in controlling autonomous camera systems, providing surgeons with high-quality surgical camera views. (2) The enhancement of voice control effects: Zinchenko et al [57] proposed a new intentional voice control method to control the long-distance motion of the robotic arm and proposed a method for voice-to-motion calibration to reduce motion ambiguity.…”

A Deep Learning Approach to Merge Rule-Based and Human-Operated Camera Control for Teleoperated Robotic Systems

“…It also provides the ability to set a dominance factor to focus more on one tool over the other [4]. Elazzazi et al extended autocamera systems to include a voice interface which allows the surgeon to vocally give commands and customize some basic system features [5]. While these systems do allow the operator to avoid context switching and focus solely on performing the surgery, they are limited by an explicit set of rules that must be hard coded into the system and often do not match what the operator would prefer.…”