Development and Validation of a Model for Laparoscopic Colorectal Surgical Instrument Recognition Using Convolutional Neural Network–Based Instance Segmentation and Videos of Laparoscopic Procedures

Abstract

IMPORTANCEDeep learning-based automatic surgical instrument recognition is an indispensable technology for surgical research and development. However, pixel-level recognition with high accuracy is required to make it suitable for surgical automation. OBJECTIVE To develop a deep learning model that can simultaneously recognize 8 types of surgical instruments frequently used in laparoscopic colorectal operations and evaluate its recognition performance. DESIGN, SETTING, AND PARTICIPANTS This quality improvement study was conducted at a single institution with a multi-institutional data set. Laparoscopic colorectal surgical videos recorded between April 1, 2009, and December 31, 2021, were included in the video data set. Deep learningbased instance segmentation, an image recognition approach that recognizes each object individually and pixel by pixel instead of roughly enclosing with a bounding box, was performed for 8 types of surgical instruments. MAIN OUTCOMES AND MEASURESAverage precision, calculated from the area under the precision-recall curve, was used as an evaluation metric. The average precision represents the number of instances of true-positive, false-positive, and false-negative results, and the mean average precision value for 8 types of surgical instruments was calculated. Five-fold cross-validation was used as the validation method. The annotation data set was split into 5 segments, of which 4 were used for training and the remainder for validation. The data set was split at the per-case level instead of the per-frame level; thus, the images extracted from an intraoperative video in the training set never appeared in the validation set. Validation was performed for all 5 validation sets, and the average mean average precision was calculated. RESULTSIn total, 337 laparoscopic colorectal surgical videos were used. Pixel-by-pixel annotation was manually performed for 81 760 labels on 38 628 static images, constituting the annotation data set. The mean average precisions of the instance segmentation for surgical instruments were 90.9% for 3 instruments, 90.3% for 4 instruments, 91.6% for 6 instruments, and 91.8% for 8 instruments. CONCLUSIONS AND RELEVANCEA deep learning-based instance segmentation model that simultaneously recognizes 8 types of surgical instruments with high accuracy was successfully developed. The accuracy was maintained even when the number of types of surgical instruments increased. This model can be applied to surgical innovations, such as intraoperative navigation and surgical automation.

Clinical evaluation of augmented reality-based 3D navigation system for brachial plexus tumor surgery

“…With the improvement of MRI technology, biomedical images of patients have become clearer and more accurate, which has made it closely related to clinical diagnosis and treatment of tumors [ 5 , 6 ]. Although MRI images can be reconstructed into 3D images, they are currently mostly presented in 2D form, and there is still a gap between conventional flat displays and the 3D structures of the human body [ 7 ].…”

Artificial Intelligence and Computer Vision during Surgery: Discussing Laparoscopic Images with ChatGPT4—Preliminary Results

“…Image recognition has been successfully applied in real-time vascular anatomical image navigation [1], or for automatically measuring distance of anatomical landmarks or size of organs intraoperatively, for example bowel length [2]. In 2022, Kitaguchi et al also reported on development of a model for laparoscopic colorectal surgical instrument recognition system using convolutional neural network-based instance segmentation and videos [3]. Real-time surgical phase recognition has been presented in many papers, using neural network-based deep learning techniques [4] [5].…”

Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling

“…In essence, instance segmentation allows for the pixelwise classification of individual objects within a surgical field, whether they are anatomical structures or surgical instruments. Although, it is the most preferred recognition technique for intraoperative guidance ( 9 ), previous attempts have been limited to segmenting rigid surgical instruments ( 10 ), as opposed to anatomical structures, which are often characterised by semi-rigid boundaries and thus pose a more difficult segmentation task.…”

“…Although few, there have been some platforms and clinical evidence in general surgery that attests to the accuracy and the significance of AI software in an intraoperative setting ( 7 , 9 , 11 – 14 ). Neurosurgery, at the forefront of cutting-edge technology, has witnessed numerous advancements in AI applications; however, these applications are limited to surgical phase recognition ( 15 ), detection and surveillance ( 16 ), diagnosis ( 17 , 18 ), endovascular navigation ( 16 ), training and preoperative planning ( 2 , 16 , 19 – 21 ), intraoperative imaging, and workflow automation ( 22 ).…”

“…In addition, Liu et al provided supporting evidence that their YOLOv3 based framework identified anatomical structures within LC more accurately than their trainees and senior surgeons (13).Laplante et al, Madani et al, and Mascagni et al, on the otherhand, utilised a semantic segmentation method to successfully determine safe and danger zones for surgical dissection and labelling anatomical structures relevant in LC(7,14). Moving forward, Kitaguchi and his team developed an instance segmentation model but they have only classified surgical instruments in laparoscopic colorectal surgeries(9). In neurosurgery, Bouget et al reports an attempt to identify intraoperative tools by an outdated method of semantic labelling and shape-based detection using supervised-vector machine (SVM) training (22), which has been further improved by Kalavakonda et al, with binary and instance segmentation approach(23).…”