Real-time vision-based surgical tool segmentation with robot kinematics prior

Su, Yun-Hsuan; Huang, Kevin; Hannaford, Blake

doi:10.1109/ismr.2018.8333305

Cited by 39 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These tasks can be categorized according the precision of the desired outputs. The finest task is tool segmentation (Bodenstedt et al, 2016 Feb-Mar;García-Peraza-Herrera et al, 2016Attia et al, 2017;Lee et al, 2017b;Zhou et al, 2017;Ross et al, 2018;Su et al, 2018). This includes multi-label tool segmentation for articulated tools (Laina et al, 2017): each tool part is associated with one label.…”

Section: Computer Vision Tasksmentioning

confidence: 99%

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Hajj¹,

Lamard²,

Conze³

et al. 2019

Medical Image Analysis

View full text Add to dashboard Cite

Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparoscopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected, the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.

show abstract

Section: Computer Vision Tasksmentioning

confidence: 99%

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Hajj¹,

Lamard²,

Conze³

et al. 2019

Medical Image Analysis

View full text Add to dashboard Cite

show abstract

“…In contrast, inexpensive commodity RGB-D cameras have provided non-contact means of collecting geometrical information and have grown in popularity in teleoperation applications [10,33,37,51]. Stereo vision is another widely used alternative, and in robot-assisted minimally invasive surgery, stereo endoscopes provide real-time 3D geometries often represented as point clouds [24,[46][47][48]. It has been shown that providing the teleoperator with depth information in the form of a real-time point cloud for certain navigation tasks can improve performance when compared to monocular RGB streams [32].…”

Section: Visual Feedback In Teleoperationmentioning

confidence: 99%

Evaluation of haptic guidance virtual fixtures and 3D visualization methods in telemanipulation—a user study

Huang

Chitrakar

Ryden

et al. 2019

Intel Serv Robotics

Self Cite

View full text Add to dashboard Cite

This work presents a user-study evaluation of various visual and haptic feedback modes on a real telemanipulation platform. Of particular interest is the potential for haptic guidance virtual fixtures and 3D-mapping techniques to enhance efficiency and awareness in a simple teleoperated valve turn task. An RGB-Depth camera is used to gather real-time color and geometric data of the remote scene, and the operator is presented with either a monocular color video stream, a 3D-mapping voxel representation of the remote scene, or the ability to place a haptic guidance virtual fixture to help complete the telemanipulation task. The efficacy of the feedback modes is then explored experimentally through a user study, and the different modes are compared on the basis of objective and subjective metrics. Despite the simplistic task and numerous evaluation metrics, results show that the haptic virtual fixture resulted in significantly better collision avoidance compared to 3D visualization alone. Anticipated performance enhancements were also observed moving from 2D to 3D visualization. Remaining comparisons lead to exploratory inferences that inform future direction for focused and statistically significant studies.

show abstract

“…Virtual fixtures can enhance task execution and shift task burdens from the operator to semiautonomous feedback. Such systems have great promise in the domains of vision-based assistance in surgical robotic procedures [14,20], joint-limit cue delivery [21], underwater robotic operations [12], integrating auxiliary sensors [22,23], and assembly and disassembly training [24].…”

Section: Haptic Feedback In Teleoperationmentioning

confidence: 99%

Telelocomotion—Remotely Operated Legged Robots

et al. 2020

Self Cite

View full text Add to dashboard Cite

Teleoperated systems enable human control of robotic proxies and are particularly amenable to inaccessible environments unsuitable for autonomy. Examples include emergency response, underwater manipulation, and robot assisted minimally invasive surgery. However, teleoperation architectures have been predominantly employed in manipulation tasks, and are thus only useful when the robot is within reach of the task. This work introduces the idea of extending teleoperation to enable online human remote control of legged robots, or telelocomotion, to traverse challenging terrain. Traversing unpredictable terrain remains a challenge for autonomous legged locomotion, as demonstrated by robots commonly falling in high-profile robotics contests. Telelocomotion can reduce the risk of mission failure by leveraging the high-level understanding of human operators to command in real-time the gaits of legged robots. In this work, a haptic telelocomotion interface was developed. Two within-user studies validate the proof-of-concept interface: (i) The first compared basic interfaces with the haptic interface for control of a simulated hexapedal robot in various levels of traversal complexity; (ii) the second presents a physical implementation and investigated the efficacy of the proposed haptic virtual fixtures. Results are promising to the use of haptic feedback for telelocomotion for complex traversal tasks.

show abstract

Real-time vision-based surgical tool segmentation with robot kinematics prior

Cited by 39 publications

References 11 publications

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Evaluation of haptic guidance virtual fixtures and 3D visualization methods in telemanipulation—a user study

Telelocomotion—Remotely Operated Legged Robots

Contact Info

Product

Resources

About