Deep learning-based anatomical site classification for upper gastrointestinal endoscopy

Abstract

Purpose Upper gastrointestinal (GI) endoscopic image documentation has provided an efficient, low-cost solution to address quality control for endoscopic reporting. The problem is, however, challenging for computer-assisted techniques, because different sites have similar appearances. Additionally, across different patients, site appearance variation may be large and inconsistent. Therefore, according to the British and modified Japanese guidelines, we propose a set of oesophagogastroduodenoscopy (EGD) images to be routinely captured and evaluate its efficiency for deep learning-based classification methods. Methods A novel EGD image dataset standardising upper GI endoscopy to several steps is established following landmarks proposed in guidelines and annotated by an expert clinician. To demonstrate the discrimination of proposed landmarks that enable the generation of an automated endoscopic report, we train several deep learning-based classification models utilising the well-annotated images. Results We report results for a clinical dataset composed of 211 patients (comprising a total of 3704 EGD images) acquired during routine upper GI endoscopic examinations. We find close agreement between predicted labels using our method and the ground truth labelled by human experts. We observe the limitation of current static image classification scheme for EGD image classification. Conclusion Our study presents a framework for developing automated EGD reports using deep learning. We demonstrate that our method is feasible to address EGD image classification and can lead towards improved performance and additionally qualitatively demonstrate its performance on our dataset.

Computer Vision in the Surgical Operating Room

“…In laparoscopic cholecystectomy, automated video retrieval can help in assessing whether a critical view of safety was achieved [36], with potential risk reduction and a safer removal of the gallbladder. Additionally, the detection and classification of anatomy enables Visc Med 2020;36:456-462 DOI: 10.1159/000511934 the automatic generation of standardized procedure reports for quality control assessment and clinical training [37].…”

Artificial intelligence in biliopancreatic endoscopy: Is there any role?

“…Utilising AI to automatically generate endoscopy reports would be a welcome solution. It has already been demonstrated that AI models can automatically identify key anatomical landmarks in luminal endoscopy, record withdrawal time, quality of bowel preparation and recognise tools [34][35][36]. Furthermore, computer vision applications can now identify phases of intervention and actions, which will likely translate to automated generation of performance measures for endoscopy quality.…”

Language-based translation and prediction of surgical navigation steps for endoscopic wayfinding assistance in minimally invasive surgery

“…Currently, image-based deep learning methods determine the endoscope's position from the classification of endoscopic images using inherent visual features [6,7]. When image labeling is provided [14], a topological scene association can be formed.…”

“…A primary benefit of this step is the simplified view of the anatomical environment since only semantic image content is addressed in the labeling of training data. Similar to [6], our approach uses the description of anatomical landmarks for the labeling of image data to simplify the task of predicting navigation steps to an association of subsequent scenes observed with content of landmarks. However, rather than focusing on spatial properties, we want to use the inherent temporal information between labeled images to extract a representation of states between images that show particular endoscope positions.…”

“…Concerning the performance, the applied image processing algorithms are highly automatable and considerably reduce the need for tracking hardware and additional imaging information in the OR [3][4][5]. The virtualization trend has recently been intensified with deep learning applications that use neural networks to classify the observed anatomy and, thereby, the position of the endoscope [6,7].…”