A Study on the VR Goggle-based Vision System for Robotic Surgery

Kim, Young Gyun; Mun, Gyeongbin; Kim, Myungjoon; Jeon, Byoungjun; Lee, Jong Hyeon; Yoon, Dan; Kim, Byeong Soo; Kong, Seong‐Ho; Jeong, Chang Wook; Lee, Kyu Eun; Cho, Minwoo; Kim, Sungwan

doi:10.1007/s12555-021-1044-6

Cited by 5 publications

(5 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It consequently clones the movement of the MTMs at the end-effector side of the instrument mounted on each PSM. To view the end-effectors’ movements remotely on PSMs, a 4-degree of freedom (DOF) endoscopic control system with a maximum resolution of 1920 × 1080 pixels, which operates as a fulcrum point motion system, is utilized for user evaluation [ 20 – 22 , 35 ]. After the processes of calibration and rectification with OpenCV libraries are completed, the stereoscopic video is displayed in the vision system using the SV or HMD.…”

Section: Methodsmentioning

confidence: 99%

“…The structure of the conventional vision system forces the operator to lower the head to see the surgical site [ 18 ]. Therefore, previous research has attempted to eliminate the restrictions on the user’s posture [ 19 – 22 ].…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, the user can have an external view without removing the VR HMD. To investigate the VVP, a user evaluation for subjects, including clinical professors, fellows, residents, and novices, was performed based on the da Vinci research kit (dVRK, Intuitive Surgical, Inc., Sunnyvale, CA, USA) in an environment similar to the operating room environment, as in previous studies [ 20 – 22 , 34 , 35 ]. By analyzing each participant’s performance results and interviews, the present study aims to demonstrate the usability of the VVP as the ergonomically improved vision system of the RAS platform.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A multimodal virtual vision platform as a next-generation vision system for a surgical robot

Kim,

Lee,

Shim

et al. 2024

Med Biol Eng Comput

Self Cite

View full text Add to dashboard Cite

Robot-assisted surgery platforms are utilized globally thanks to their stereoscopic vision systems and enhanced functional assistance. However, the necessity of ergonomic improvement for their use by surgeons has been increased. In surgical robots, issues with chronic fatigue exist owing to the fixed posture of the conventional stereo viewer (SV) vision system. A head-mounted display was adopted to alleviate the inconvenience, and a virtual vision platform (VVP) is proposed in this study. The VVP can provide various critical data, including medical images, vital signs, and patient records, in three-dimensional virtual reality space so that users can access medical information simultaneously. An availability of the VVP was investigated based on various user evaluations by surgeons and novices, who executed the given tasks and answered questionnaires. The performances of the SV and VVP were not significantly different; however, the craniovertebral angle of the VVP was 16.35° higher on average than that of the SV. Survey results regarding the VVP were positive; participants indicated that the optimal number of displays was six, preferring the 2 × 3 array. Reflecting the tendencies, the VVP can be a neoconceptual candidate to be customized for medical use, which opens a new prospect in a next-generation surgical robot. Graphical Abstract

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A multimodal virtual vision platform as a next-generation vision system for a surgical robot

Kim,

Lee,

Shim

et al. 2024

Med Biol Eng Comput

Self Cite

View full text Add to dashboard Cite

show abstract

“…These are dynamically mapped to the user's control interface and virtual environment. Other studies combine VR interfaces with haptic devices to provide an enhanced environment [17]. For example, Wang et al [38] introduced a remote robot welding system using VR headsets as interactive devices.…”

Section: Vr Robotics Interfacementioning

confidence: 99%

“…In the present era, human interaction methods [16] have undergone a transformation, transitioning from traditional 2D interactions to more efficient and intuitively immersive 3D interaction approaches. In light of this evolution, VR technology has assumed a significant role in next-generation intelligent HRC [17][18][19][20][21]. VR has the capacity to create fully immersive virtual spaces, accessible only through VR devices, offering an enhanced, intuitive visualization of the created virtual environments.…”

Section: Introductionmentioning

confidence: 99%

Research on Teleoperated Virtual Reality Human–Robot Five-Dimensional Collaboration System

Zhang,

Liu,

Duan

et al. 2023

Biomimetics

View full text Add to dashboard Cite

In the realm of industrial robotics, there is a growing challenge in simplifying human–robot collaboration (HRC), particularly in complex settings. The demand for more intuitive teleoperation systems is on the rise. However, optimizing robot control interfaces and streamlining teleoperation remains a formidable task due to the need for operators to possess specialized knowledge and the limitations of traditional methods regarding operational space and time constraints. This study addresses these issues by introducing a virtual reality (VR) HRC system with five-dimensional capabilities. Key advantages of our approach include: (1) real-time observation of robot work, whereby operators can seamlessly monitor the robot’s real-time work environment and motion during teleoperation; (2) leveraging VR device capabilities, whereby the strengths of VR devices are harnessed to simplify robot motion control, significantly reducing the learning time for operators; and (3) adaptability across platforms and environments: our system effortlessly adapts to various platforms and working conditions, ensuring versatility across different terminals and scenarios. This system represents a significant advancement in addressing the challenges of HRC, offering improved teleoperation, simplified control, and enhanced accessibility, particularly for operators with limited prior exposure to robot operation. It elevates the overall HRC experience in complex scenarios.

show abstract

Unconstrained lightweight control interface for robot-assisted minimally invasive surgery using MediaPipe framework and head-mounted display

Rhee,

Kim,

Lee

et al. 2024

Virtual Reality

View full text Add to dashboard Cite

Robotic surgery is preferred over open or laparoscopic surgeries due to its intuitiveness and convenience. However, prolonged use of surgical robots can cause neck pain and joint fatigue in wrist and fingers. Also, input systems are bulky and difficult to maintain. To resolve these issues, we propose a novel input module based on real-time 3D hand tracking driven by RGB images and MediaPipe framework to control surgical robots such as patient side manipulator (PSM) and endoscopic camera manipulator (ECM) of da Vinci research kit. In this paper, we explore the mathematical basis of the proposed 3D hand tracking module and provide a proof-of-concept through user experience (UX) studies conducted in a virtual environment. End-to-end latencies for controlling PSM and ECM were 170 ± 10 ms and 270 ± 10 ms, respectively. Of fifteen novice participants recruited for the UX study, thirteen managed to reach a qualifiable level of proficiency after 50 min of practice and fatigue of hand and wrist were imperceivable. Therefore, we concluded that we have successfully developed a robust 3D hand tracking module for surgical robot control and in the future, it would hopefully reduce hardware cost and volume as well as resolve ergonomic problems. Furthermore, RGB image driven 3D hand tracking module developed in our study can be widely applicable to diverse fields such as extended reality (XR) development and remote robot control. In addition, we provide a new standard for evaluating novel input modalities of XR environments from a UX perspective.

show abstract

A Study on the VR Goggle-based Vision System for Robotic Surgery

Cited by 5 publications

References 65 publications

A multimodal virtual vision platform as a next-generation vision system for a surgical robot

A multimodal virtual vision platform as a next-generation vision system for a surgical robot

Research on Teleoperated Virtual Reality Human–Robot Five-Dimensional Collaboration System

Unconstrained lightweight control interface for robot-assisted minimally invasive surgery using MediaPipe framework and head-mounted display

Contact Info

Product

Resources

About