Interactive and Multimodal-based Augmented Reality for Remote Assistance using a Digital Surgical Microscope

Wisotzky, Eric L.; Rosenthal, Jean-Claude; Eisert, Peter; Hilsmann, Anna; Schmid, Falko; Bauer, Michael; Schneider, Armin; Uecker, Florian C.

doi:10.1109/vr.2019.8797682

Cited by 19 publications

(17 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These differences in the optical parameters substantiate the validity to detect and identify, automatically or semiautomatically, tissue types for active treatment decisions during imageguided surgery. 2,25,26 Intraoperative tissue properties can be analyzed using multispectral imaging to point out tissue differences in the visualization process using the knowledge acquired in this study. By contrast, it seems possible to use the results presented here to learn a classifier that makes tissue types more distinguishable intraoperatively.…”

Section: Resultsmentioning

confidence: 99%

Determination of optical properties of human tissues obtained from parotidectomy in the spectral range of 250 to 800 nm

et al. 2019

Self Cite

View full text Add to dashboard Cite

The optical properties of human tissues are an important parameter in medical diagnostics and therapy. The knowledge of these parameters can encourage the development of automated, computer-driven optical tissue analysis methods. We determine the absorption coefficient μ a and scattering coefficient μ 0 s of different tissue types obtained during parotidectomy in the wavelength range of 250 to 800 nm. These values are determined by high precision integrating sphere measurements in combination with an optimized inverse Monte Carlo simulation. To conserve the optical behavior of living tissues, the optical spectroscopy measurements are performed immediately after tissue removal. Our study includes fresh samples of the ear, nose, and throat (ENT) region, as muscle tissue, nervous tissue, white adipose tissue, stromal tissue, parotid gland, and tumorous tissue of five patients. The measured behavior of adipose corresponds well with the literature, which sustains the applied method. It is shown that muscle is well supplied with blood as it features the same characteristic peaks at 430 and 555 nm in the absorption curve. The parameter μ 0 s decreases for all tissue types above 570 nm. The accuracy is adequate for the purposes of providing μ a and μ 0 s of different human tissue types as muscle, fat, nerve, or gland tissue, which are embedded in large complex structures such as in the ENT area. It becomes possible for the first time to present reasonable results for the optical behavior of human soft tissue located in the ENT area and in the near-UV, visual, and near-infrared areas. © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

show abstract

Section: Resultsmentioning

confidence: 99%

Determination of optical properties of human tissues obtained from parotidectomy in the spectral range of 250 to 800 nm

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Thus, the main analysis is done only on the blue channel of the sensor, and all spectral bands are considered using the illumination

within the defined spectral range. A similar approach was proposed in Wisotzky et al [ 18 ] using a fully digital surgical microscope. In this study, the system is used to acquire all stereo-spectral information by rotating the filter-wheel.…”

Section: Methodsmentioning

confidence: 99%

“…All this additional knowledge has the potential to support the surgeon in his/her decision making and facilitate the surgical procedure when appropriate intraoperative real-time visualizations are used. Thus, these new image-based methods have the potential to accelerate complex surgical procedures and avoid re-surgery for an improved patient outcome [ 18 ].…”

Section: Introductionmentioning

confidence: 99%

Surgical Guidance for Removal of Cholesteatoma Using a Multispectral 3D-Endoscope

Wisotzky

Rosenthal

Wege

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

We develop a stereo-multispectral endoscopic prototype in which a filter-wheel is used for surgical guidance to remove cholesteatoma tissue in the middle ear. Cholesteatoma is a destructive proliferating tissue. The only treatment for this disease is surgery. Removal is a very demanding task, even for experienced surgeons. It is very difficult to distinguish between bone and cholesteatoma. In addition, it can even reoccur if not all tissue particles of the cholesteatoma are removed, which leads to undesirable follow-up operations. Therefore, we propose an image-based method that combines multispectral tissue classification and 3D reconstruction to identify all parts of the removed tissue and determine their metric dimensions intraoperatively. The designed multispectral filter-wheel 3D-endoscope prototype can switch between narrow-band spectral and broad-band white illumination, which is technically evaluated in terms of optical system properties. Further, it is tested and evaluated on three patients. The wavelengths 400 nm and 420 nm are identified as most suitable for the differentiation task. The stereoscopic image acquisition allows accurate 3D surface reconstruction of the enhanced image information. The first results are promising, as the cholesteatoma can be easily highlighted, correctly identified, and visualized as a true-to-scale 3D model showing the patient-specific anatomy.

show abstract

“…18 These aspects make a robust calibration process necessary for potential clinical practice. 19,20 This work presents a calibration pipeline for different possible hyperspectral setup configurations addressing the above-mentioned challenges. We describe two approaches for intraoperative image-guided soft tissue analysis, the first selects the wavelengths on the illumination side and the second on the sensor side and introduce our robust calibration pipeline, which makes both systems comparable to each other.…”

Section: Introductionmentioning

confidence: 99%

Validation of two techniques for intraoperative hyperspectral human tissue determination

et al. 2020

Self Cite

View full text Add to dashboard Cite

Purpose: Hyperspectral imaging (HSI) is a non-contact optical imaging technique with the potential to serve as an intraoperative computer-aided diagnostic tool. Our work analyzes the optical properties of visible structures in the surgical field for automatic tissue categorization. Approach: Building an HSI-based computer-aided tissue analysis system requires accurate ground truth and validation of optical soft tissue properties as these show large variability. We introduce and validate two different hyperspectral intraoperative imaging setups and their use for the analysis of optical tissue properties. First, we present an improved multispectral filterwheel setup integrated into a fully digital microscope. Second, we present a novel setup of two hyperspectral snapshot cameras for intraoperative usage. Both setups are operating in the spectral range of 400 up to 975 nm. They are calibrated and validated using the same database and calibration set. Results: For validation, a color chart with 18 well-defined color spectra in the visual range is analyzed. Thus the results acquired with both settings become transferable and comparable to each other as well as between different interventions. On patient data of two different otorhinolaryngology procedures, we analyze the optical behaviors of different soft tissues and show a visualization of such different spectral information. Conclusion: The introduced calibration pipeline for different HSI setups allows comparison between all acquired spectral information. Clinical in vivo data underline the potential of HSI as an intraoperative diagnostic tool and the clinical usability of both introduced setups. Thereby, we demonstrate their feasibility for the in vivo analysis and categorization of different human soft tissues.

show abstract

Interactive and Multimodal-based Augmented Reality for Remote Assistance using a Digital Surgical Microscope

Cited by 19 publications

References 29 publications

Determination of optical properties of human tissues obtained from parotidectomy in the spectral range of 250 to 800 nm

Determination of optical properties of human tissues obtained from parotidectomy in the spectral range of 250 to 800 nm

Surgical Guidance for Removal of Cholesteatoma Using a Multispectral 3D-Endoscope

Validation of two techniques for intraoperative hyperspectral human tissue determination

Contact Info

Product

Resources

About