Surgical skills assessment from robot assisted surgery video data

Yu, Ming; Yang, Cheng; Yuan, Jing; Li, Liangzhe; Yang, Pengcheng; Zhang, Guang; Feng, Chen

doi:10.1109/icpeca51329.2021.9362525

Cited by 6 publications

(12 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our generated data combined with the benchmark of Anh et al [ 34 ] successfully outperformed the solutions of Ming et al [ 26 ] in both Suturing and Knot-Tying, and they only slightly fell short in their results in Needle-Passing.…”

Section: Resultsmentioning

confidence: 87%

“…Ming et al found Knot-Tying to be the easiest surgical task to assess with both STIP and iDT [ 26 ]. Our method is similar in principle to their STIP method, as it also tracks the movement of interest points/features over time.…”

Section: Resultsmentioning

confidence: 99%

“…The same observations apply to Suturing as to Knot-Tying: ResNet, CNN + LSTM and convAuto performed well; LSTM showed high results in some cases, accompanied by confusion matrix anomalies (the maximum expert recall only being 3.33%, with an overall average of 0.22%); CNN seemingly performed the worst, even though it still did so above 75% on model average accuracies. Even though Ming et al [ 26 ] also found that Suturing is harder to classify than Knot-Tying, CNN was found to be one of the most reliable methods by Yanik et al in their review [ 21 ]. It is possible that the CNN model of the benchmark is too small, and it would perform better with higher complexity and more layers.…”

Section: Resultsmentioning

confidence: 99%

“…Ming et al claimed that Needle-Passing was the hardest skill to perform classification for, since they did not find significant differences between the trajectories of expert and novice users’ left hand movements [ 26 ]. With our data generation method, only LSTM dropped significantly in efficacy in comparison with its performance on the other skills.…”

Section: Resultsmentioning

confidence: 99%

“…Ming et al achieved a mean accuracy of 79.29%/76.79%, 80.71%/83.81% and 72.57% /76.65% on the basis of Space Temporal Interest Points (STIP)/Improved Dense Trajectory (iDT) representation of the three subtasks of JIGSAWS, respectively [ 26 ]. The STIP method is built on similar theoretical bases as our sparse optical flow-based solution, given that both of them track the movement of points over time.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Endoscopic Image-Based Skill Assessment in Robot-Assisted Minimally Invasive Surgery

Lajkó

Elek

Haidegger

2021

Sensors

View full text Add to dashboard Cite

Objective skill assessment-based personal performance feedback is a vital part of surgical training. Either kinematic—acquired through surgical robotic systems, mounted sensors on tooltips or wearable sensors—or visual input data can be employed to perform objective algorithm-driven skill assessment. Kinematic data have been successfully linked with the expertise of surgeons performing Robot-Assisted Minimally Invasive Surgery (RAMIS) procedures, but for traditional, manual Minimally Invasive Surgery (MIS), they are not readily available as a method. 3D visual features-based evaluation methods tend to outperform 2D methods, but their utility is limited and not suited to MIS training, therefore our proposed solution relies on 2D features. The application of additional sensors potentially enhances the performance of either approach. This paper introduces a general 2D image-based solution that enables the creation and application of surgical skill assessment in any training environment. The 2D features were processed using the feature extraction techniques of a previously published benchmark to assess the attainable accuracy. We relied on the JHU–ISI Gesture and Skill Assessment Working Set dataset—co-developed by the Johns Hopkins University and Intuitive Surgical Inc. Using this well-established set gives us the opportunity to comparatively evaluate different feature extraction techniques. The algorithm reached up to 95.74% accuracy in individual trials. The highest mean accuracy—averaged over five cross-validation trials—for the surgical subtask of Knot-Tying was 83.54%, for Needle-Passing 84.23% and for Suturing 81.58%. The proposed method measured well against the state of the art in 2D visual-based skill assessment, with more than 80% accuracy for all three surgical subtasks available in JIGSAWS (Knot-Tying, Suturing and Needle-Passing). By introducing new visual features—such as image-based orientation and image-based collision detection—or, from the evaluation side, utilising other Support Vector Machine kernel methods, tuning the hyperparameters or using other classification methods (e.g., the boosted trees algorithm) instead, classification accuracy can be further improved. We showed the potential use of optical flow as an input for RAMIS skill assessment, highlighting the maximum accuracy achievable with these data by evaluating it with an established skill assessment benchmark, by evaluating its methods independently. The highest performing method, the Residual Neural Network, reached means of 81.89%, 84.23% and 83.54% accuracy for the skills of Suturing, Needle-Passing and Knot-Tying, respectively.

show abstract

Section: Resultsmentioning

confidence: 87%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Endoscopic Image-Based Skill Assessment in Robot-Assisted Minimally Invasive Surgery

Lajkó

Elek

Haidegger

2021

Sensors

View full text Add to dashboard Cite

show abstract

Video-based skill acquisition assessment in laparoscopic surgery using deep learning

Yanik,

Ainam,

et al. 2024

Global Surg Educ

View full text Add to dashboard Cite

Deep Learning Based Image Processing for Robot Assisted Surgery: A Systematic Literature Survey

et al. 2022

View full text Add to dashboard Cite

The recent advancements in the surging field of Deep Learning (DL) have revolutionized every sphere of life, and the healthcare domain is no exception. The enormous success of DL models, particularly with image data, has led to the development of image-guided Robot Assisted Surgery (RAS) systems. By and large, the number of studies concerning image-driven computer assisted surgical systems using DL has increased exponentially. Additionally, the contemporary availability of surgical datasets has also boosted the DL applications in RAS. Inspired by the latest trends and contributions in surgery, this literature survey presents a summarized analysis of recent innovations of DL in image-guided RAS systems. After a thorough review, a sum of 184 articles are selected and grouped into four categories, based on the literature and the relevancy of the task in the articles, comprising 1) Surgical Tools, 2) Surgical Processes, 3) Surgical Surveillance, and 4) Surgical Performance. The survey also discusses publicly available surgical datasets and highlights the basics of the DL models. Furthermore, the legal, ethical, and technological challenges together with the intuitive predictions and recommendations related to the autonomous RAS systems are also presented. The study reveals that Convolutional Neural Network (CNN) is most widely adopted architecture, whereas, JIGSAWS is most employed dataset in RAS. The study suggests fusing kinematic data along with image data, which produces better accuracy and precision, particularly in gesture and trajectory segmentation tasks. Additionally, CNN and Long Short Term Memory (LSTM) networks have shown remarkable performance, however, authors recommend employing these gigantic architectures only when simpler models have failed to produce satisfactory results. The simpler models, despite their limitations, are time and cost effective and yield considerable outcomes even on the smaller datasets.

show abstract

Surgical skills assessment from robot assisted surgery video data

Cited by 6 publications

References 15 publications

Endoscopic Image-Based Skill Assessment in Robot-Assisted Minimally Invasive Surgery

Endoscopic Image-Based Skill Assessment in Robot-Assisted Minimally Invasive Surgery

Video-based skill acquisition assessment in laparoscopic surgery using deep learning

Deep Learning Based Image Processing for Robot Assisted Surgery: A Systematic Literature Survey

Contact Info

Product

Resources

About