Multiple instance convolutional neural network for gallbladder assessment from laparoscopic images

Abstract

Background We present an artificial intelligence framework for vascularity classification of the gallbladder (GB) wall from intraoperative images of laparoscopic cholecystectomy (LC). Methods A two‐stage Multiple Instance Convolutional Neural Network is proposed. First, a convolutional autoencoder is trained to extract feature representations from 4585 patches of GB images. The second model includes a multi‐instance encoder that fetches random patches from a GB region and outputs an equal number of embeddings that feed a multi‐input classification module, which employs pooling and self‐attention mechanisms, to perform prediction. Results The evaluation was performed on 234 GB images of low and high vascularity from 68 LC videos. Thorough comparison with various state‐of‐the‐art multi‐instance and single‐instance learning algorithms was performed for two experimental tasks: image‐ and video‐level classification. The proposed framework shows the best performance with accuracy 92.6%–93.2% and F1 93.5%–93.9%, close to the agreement of two expert evaluators (94%). Conclusions The proposed technique provides a novel approach to classify LC operations with respect to the vascular pattern of the GB wall.

Prediction of remaining surgery duration in laparoscopic videos based on visual saliency and the transformer network

“…Although this type of information was not available in the dataset, it may well be used as an additional input in the RSD pipeline. For example, prior studies have shown that important information about the GB condition, which is related to the operation complexity, may well be extracted from still images captured during the operation 32 . In the future, we aim to fuse the proposed model with such intraoperative indicators and preoperative data to improve RSD prediction.…”