CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection

Nwoye, Chinedu Innocent; Yu, Tong; Sharma, Saurav; Murali, Aditya; Alapatt, Deepak; Vardazaryan, Armine; Yuan, Kun; Hajek, Jonas; Reiter, Wolfgang; Yamlahi, Amine; Smidt, Finn-Henri; Zou, Xiaoyang; Zheng, Guoyan; Oliveira, Bruno; Torres, Helena R.; Kondo, Satoshi; Kasai, S.; Holm, Felix; Özsoy, Ege; Gui, Shuangchun; Li, Han; Raviteja, Sista; Sathish, Rachana; Poudel, Pranav; Bhattarai, Binod; Wang, Ziheng; Guo, Rui; Schellenberg, Melanie; Vilaça, João L.; Czempiel, Tobias; Wang, Zhenkun; Sheet, Debdoot; Thapa, Shrawan Kumar; Berniker, Max; Godau, Patrick; Morais, Pedro; Regmi, Sudarshan; Tran, Thuy Nuong; Fonseca, Jaime C.; Nölke, Jan-Hinrich; Lima, Estêvão; Vazquez, Eduard; Maier‐Hein, Klaus H.; Navab, Nassir; Mascagni, Pietro; Seeliger, Barbara; González, Cristians; Mutter, Didier; Padoy, Nicolas

doi:10.1016/j.media.2023.102888

Cited by 9 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the second experiment, the algorithm cross-validation mAP score using our own HeiCholeActivity dataset as well as CholecT45 as input improved over the score achieved by the algorithm using only CholecT45 as input (37.9% vs. 36.5%) [29,46]. This improvement is encouraging for further research on state-of-the-art algorithms which achieve 40.6% mAP using only CholecT45 for training [53].…”

Section: Activity Recognition With MLmentioning

confidence: 79%

“…In the first experiment, the algorithm's cross-validation mAP score using all of our own HeiCholeActivity dataset as input (25.7%) was lower than during its participation in the CholecTriplet2022 challenge which used the CholecT45 dataset as input (36.5%) [29]. Possible reasons for this difference in performance can be addressed.…”

Section: Activity Recognition With MLmentioning

confidence: 93%

“…A machine learning (ML) algorithm was trained and tested for automatic surgical action recognition on the annotated data. A ML algorithm called Distilled-Swin, which ranked first in the action triplet recognition category of the MIC-CAI EndoVis challenge CholecTriplet2022, the variant (SwinT + MultiD + SelfD) model, was used [29,46]. Distilled-Swin is a swin transformer model trained through self-distillation, this approach is specifically designed to address challenges related to high number of classes and class imbalance [47].…”

Section: Data Processingmentioning

confidence: 99%

“…To train machine learning (ML) algorithms with surgical knowledge for fine-granular surgical workflow recognition, surgical workflow information needs to be captured and structured in large quantity but data which includes fine-granular surgical workflow annotation for LC is lacking [23][24][25]. One established dataset, CholecT45 (previously CholecT40, based on videos from the single-center Cholec80 benchmark dataset), uses the concept of action triplets (instrument, verb, target) for surgical action data annotation and was used as a benchmark dataset for surgical workflow recognition in the MICCAI EndoVis challenges CholecTriplet2021 and CholecTriplet2022 [26][27][28][29][30]. However, clear action, verb and target class annotation rules are not publicly available and the used annotation software is proprietary, thus the dataset is neither reproducible nor easily extendable [31,32].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

Younis,

Yamlahi,

Bodenstedt

et al. 2024

Surg Endosc

View full text Add to dashboard Cite

Background Laparoscopic cholecystectomy is a very frequent surgical procedure. However, in an ageing society, less surgical staff will need to perform surgery on patients. Collaborative surgical robots (cobots) could address surgical staff shortages and workload. To achieve context-awareness for surgeon-robot collaboration, the intraoperative action workflow recognition is a key challenge. Methods A surgical process model was developed for intraoperative surgical activities including actor, instrument, action and target in laparoscopic cholecystectomy (excluding camera guidance). These activities, as well as instrument presence and surgical phases were annotated in videos of laparoscopic cholecystectomy performed on human patients (n = 10) and on explanted porcine livers (n = 10). The machine learning algorithm Distilled-Swin was trained on our own annotated dataset and the CholecT45 dataset. The validation of the model was conducted using a fivefold cross-validation approach. Results In total, 22,351 activities were annotated with a cumulative duration of 24.9 h of video segments. The machine learning algorithm trained and validated on our own dataset scored a mean average precision (mAP) of 25.7% and a top K = 5 accuracy of 85.3%. With training and validation on our dataset and CholecT45, the algorithm scored a mAP of 37.9%. Conclusions An activity model was developed and applied for the fine-granular annotation of laparoscopic cholecystectomies in two surgical settings. A machine recognition algorithm trained on our own annotated dataset and CholecT45 achieved a higher performance than training only on CholecT45 and can recognize frequently occurring activities well, but not infrequent activities. The analysis of an annotated dataset allowed for the quantification of the potential of collaborative surgical robots to address the workload of surgical staff. If collaborative surgical robots could grasp and hold tissue, up to 83.5% of the assistant’s tissue interacting tasks (i.e. excluding camera guidance) could be performed by robots.

show abstract

Section: Activity Recognition With MLmentioning

confidence: 79%

Section: Activity Recognition With MLmentioning

confidence: 93%

Section: Data Processingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

Younis,

Yamlahi,

Bodenstedt

et al. 2024

Surg Endosc

View full text Add to dashboard Cite

show abstract

“…Recent studies that apply machine learning algorithms to this task have shown highly promising results [ 2 , 3 ]. To further support advances in this area, academic machine learning competitions are hosted regularly [ 4 – 6 ]. However, despite the progress in surgical workflow recognition, the developers of machine learning algorithms are faced with several challenges that result from the heterogeneous nature and complexity of surgical workflows, as well as the temporal correlation of sensor data.…”

Section: Introductionmentioning

confidence: 99%

Surgical phase and instrument recognition: how to identify appropriate dataset splits

Kostiuchik,

Sharan,

Mayer

et al. 2024

Int J CARS

View full text Add to dashboard Cite

Purpose Machine learning approaches can only be reliably evaluated if training, validation, and test data splits are representative and not affected by the absence of classes. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, sub-properties like instrument (co-)occurrence are usually not particularly considered when defining the split. Methods We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits, especially regarding identification of sub-optimal dataset splits. Results We performed analysis of the datasets Cholec80, CATARACTS, CaDIS, M2CAI-workflow, and M2CAI-tool using the proposed application. We were able to uncover phase transitions, individual instruments, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements in the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks. Conclusion In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split because it can greatly influence the assessments of machine learning approaches. Our interactive tool allows for determination of better splits to improve current practices in the field. The live application is available at https://cardio-ai.github.io/endovis-ml/.

show abstract

Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Sharma,

Nwoye,

Mutter

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection

Cited by 9 publications

References 30 publications

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

Surgical phase and instrument recognition: how to identify appropriate dataset splits

Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Contact Info

Product

Resources

About