On mixed reality environments for minimally invasive therapy guidance: Systems architecture, successes and challenges in their implementation from laboratory to clinic

Abstract

Mixed reality environments for medical applications have been explored and developed over the past three decades in an effort to enhance the clinician’s view of anatomy and facilitate the performance of minimally invasive procedures. These environments must faithfully represent the real surgical field and require seamless integration of pre- and intra-operative imaging, surgical instrument tracking, and display technology into a common framework centered around and registered to the patient. However, in spite of their reported benefits, few mixed reality environments have been successfully translated into clinical use. Several challenges that contribute to the difficulty in integrating such environments into clinical practice are presented here and discussed in terms of both technical and clinical limitations. This article should raise awareness among both developers and end-users toward facilitating a greater application of such environments in the surgical practice of the future.

See It With Your Own Eyes: Markerless Mobile Augmented Reality for Radiation Awareness in the Hybrid Room

“…For an AR system to have a real impact on clinical outcome, it should not only achieve accurate registration and tracking, robust visualization and convincing display, but also fit smoothly within the OR [4]. The last decades have seen the development of several solutions for medical AR looking to cope with these challenges.…”

“…The last decades have seen the development of several solutions for medical AR looking to cope with these challenges. We limit our study to a subset of them but we refer the reader to [4], [16], [17] for a detailed literature review on medical AR systems.…”

“…However, permission to use this material for any other purposes must be obtained from the IEEE by sending an email to pubs-permissions@ieee.org. registration of the viewer's point-of-view, necessary for the AR visualization, while interfering the least possible with the clinical workflow [4]. To cope with this challenge, current systems rely mostly on tracking markers placed in the scene or on the surgical tools [5]- [8].…”

Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy

“…At this point, the assistance system is no longer useful. To realize the "Intelligence Amplification" and the potential of computerized surgery, context-aware information filtering is necessary to reduce the amount of presented data [5,6].…”

Image-guided intraoperative radiation therapy: current developments and future perspectives

“…Accordingly [14], the system that is best placed to have an impact on clinical outcome will offer not only accurate registration and tracking, but also seamless integration within the standard operating room and no interference with the traditional clinical workflow. The multi-camera OTS complies with some of these desired features in a truly challenging clinical task as IOERT.…”