Software Framework for Customized Augmented Reality Headsets in Medicine

Abstract

The growing availability of self-contained and affordable augmented reality headsets such as the Microsoft HoloLens is encouraging the adoption of these devices also in the healthcare sector. However, technological and human-factor limitations still hinder their routine use in clinical practice. Among them, the major drawbacks are due to their general-purpose nature and to the lack of a standardized framework suited for medical applications and devoid of platform-dependent tracking techniques and/or complex calibration procedures. To overcome such limitations, in this paper we present a software framework that is designed to support the development of augmented reality applications for custom-made headmounted displays designed to aid high-precision manual tasks. The software platform is highly configurable, computationally efficient, and it allows the deployment of augmented reality applications capable to support in situ visualization of medical imaging data. The framework can provide both optical and video see-throughbased augmentations and it features a robust optical tracking algorithm. An experimental study was designed to assess the efficacy of the platform in guiding a simulated task of surgical incision. In the experiments, the user was asked to perform a digital incision task, with and without the aid of the augmented reality headset. The task accuracy was evaluated by measuring the similarity between the traced curve and the planned one. The average error in the augmented reality tests was < 1 mm. The results confirm that the proposed framework coupled with the new-concept headset may boost the integration of augmented reality headsets into routine clinical practice.

Wearable Augmented Reality Platform for Aiding Complex 3D Trajectory Tracing

“…In this study, only a qualitative assessment of latency and spatial jitter/drift due to inaccuracies in the inside-out tracking and to the VST mechanism was performed. However, considering the results obtained with this and previous studies [23,31,32], we can reasonably argue that the proposed wearable VST approach is adequate in ensuring a stable VST AR guidance for manual tasks that demand high accuracy and for which the subject can compensate for display latency by working more slowly.…”

“…As described in [23], the key features of the HMD were established with the aim of mitigating relevant perceptual conflicts typical of commercial AR headsets for close-up activities.…”

“…As previously mentioned, the accurate AR overlay of the virtual trajectory to the physical 3D-printed models is achieved by means of a tracking modality that relies on the real-time localization of three reference markers; for this reason, three spherical markers (11 mm in diameter) were embedded in the CAD model of phantoms as shown in Figure 2. The markers were dyed in fluorescent green, to boost the response of the camera sensor and improve the robustness of the blob detection under uncontrolled lighting conditions [23,29].…”

Ambiguity-Free Optical–Inertial Tracking for Augmented Reality Headsets

“…The primary goal of visual augmented reality (AR) technology is to enrich the visual perception of the surrounding space by overlaying three-dimensional (3D) computer-generated elements on it in a spatially realistic manner. In any AR application, the problem of correctly defining the spatial location of the digital elements with respect to the real scene is the principal factor that provides the user with a sense of perceptive congruity [1,2]. In order to satisfy the locational realism of the AR view and achieve an accurate spatial alignment between real-world scene and virtual elements, the process of image formation of the virtual content must be the same of the real-world scene [3].…”

“…The algorithm was tested on a poorly-calibrated custom-made see-through HMD operating under uncontrolled lighting conditions, specifically conceived for high-precision manual tasks in a surgical or industrial scenario, and capable of providing both video and optical see-through-based augmentations [2].…”