Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks

Abstract

Surgical guidance and decision making could be improved with accurate and real-time measurement of intra-operative data including shape and spectral information of the tissue surface. In this work, a dual-modality endoscopic system has been proposed to enable tissue surface shape reconstruction and hyperspectral imaging (HSI). This system centers around a probe comprised of an incoherent fiber bundle, whose fiber arrangement is different at the two ends, and miniature imaging optics. For 3D reconstruction with structured light (SL), a light pattern formed of randomly distributed spots with different colors is projected onto the tissue surface, creating artificial texture. Pattern decoding with a Convolutional Neural Network (CNN) model and a customized feature descriptor enables real-time 3D surface reconstruction at approximately 12 frames per second (FPS). In HSI mode, spatially sparse hyperspectral signals from the tissue surface can be captured with a slit hyperspectral imager in a single snapshot. A CNN based super-resolution model, namely "super-spectral-resolution" network (SSRNet), has also been developed to estimate pixel-level dense hypercubes from the endoscope cameras standard RGB images and the sparse hyperspectral signals, at approximately 2 FPS. The probe, with a 2.1 mm diameter, enables the system to be used with endoscope working channels. Furthermore, since data acquisition in both modes can be accomplished in one snapshot, operation of this system in clinical applications is minimally affected by tissue surface movement and deformation. The whole apparatus has been validated on phantoms and tissue (ex vivo and in vivo), while initial measurements on patients during laryngeal surgery show its potential in real-world clinical applications.

Hyperspectral Imaging and the Retina: Worth the Wave?

“…Examples of HSI usability within (bio)medical disciplines range from perioperative support with guidance of the surgeon to delineate the right resection margins of lentigo maligna or cerebral neoplasms to assessing diabetic foot ulcer development risk. [2][3][4][5][6] Other proof-of-concepts measured the oxygen saturation (OS) of various organs 7 ; assessed the presence of molecules such as cholesterol, melanin, and hemoglobin 8,9 ; enhanced the surgeon's vision in oncologic surgery and laparoscopy [10][11][12][13] ; predicted hemorrhagic shock and appraising hemodynamics [14][15][16] ; classified corneal injury 17 ; augmented contrast for histologic examinations 18,19 ; and detected neoplasms of the skin, mouth, colon, brain, and others. 3,5,[20][21][22] In ophthalmology, HSI can be used to assess the state and distribution of chromophores, such as cytochrome C, and assess the metabolic status of hemoglobin in the context of retinal blood vessel oxygenation.…”

A background correction method to compensate illumination variation in hyperspectral imaging

“…HSI has shown potential in a range of biomedical applications, from label-free tumour diagnoses [4][5][6] and detection of tumour margins during surgical operations [7][8][9], to quantification of blood oxygenation levels [10][11][12], and multi-colour fluorescent imaging [12,13]. HSI methods have thus been developed for the fast and accurate analysis of biological samples ex vivo [14][15][16][17] as well as for diagnostic and intraoperative applications in vivo [16,18].…”

Deep learning applied to hyperspectral endoscopy for online spectral classification

“…However, due to SVMs being binary classifiers, their speed is poor when used in the online segmentation of multi-class problems. Finally, advanced learning algorithms have been shown recently to combine speed with accuracy [13][14][15][16][17] , which makes them promising candidates for online evaluation of HSI data. Such algorithms have seen a wide range of applications in the field of hyperspectral imaging of tissue, from the prediction of spectral signals from white light images 14 to the extraction of specific measures of cancer progression 14,16 .…”

“…Finally, advanced learning algorithms have been shown recently to combine speed with accuracy [13][14][15][16][17] , which makes them promising candidates for online evaluation of HSI data. Such algorithms have seen a wide range of applications in the field of hyperspectral imaging of tissue, from the prediction of spectral signals from white light images 14 to the extraction of specific measures of cancer progression 14,16 . For direct image interpretation, techniques such as generative adversarial networks 15 and fully-convolutional neural networks 15,17 have achieved success, with pixel-wise classifiers also showing high performance 13 .…”