UltrARsound: in situ visualization of live ultrasound images using HoloLens 2

Abstract

Purpose Augmented Reality (AR) has the potential to simplify ultrasound (US) examinations which usually require a skilled and experienced sonographer to mentally align narrow 2D cross-sectional US images in the 3D anatomy of the patient. This work describes and evaluates a novel approach to track retroreflective spheres attached to the US probe using an inside-out technique with the AR glasses HoloLens 2. Finally, live US images are displayed in situ on the imaged anatomy. Methods The Unity application UltrARsound performs spatial tracking of the US probe and attached retroreflective markers using the depth camera integrated into the AR glasses—thus eliminating the need for an external tracking system. Additionally, a Kalman filter is implemented to improve the noisy measurements of the camera. US images are streamed wirelessly via the PLUS toolkit to HoloLens 2. The technical evaluation comprises static and dynamic tracking accuracy, frequency and latency of displayed images. Results Tracking is performed with a median accuracy of 1.98 mm/1.81$$^\circ $$ ∘ for the static setting when using the Kalman filter. In a dynamic scenario, the median error was 2.81 mm/1.70$$^\circ $$ ∘ . The tracking frequency is currently limited to 20 Hz. 83% of the displayed US images had a latency lower than 16 ms. Conclusions In this work, we showed that spatial tracking of retroreflective spheres with the depth camera of HoloLens 2 is feasible, achieving a promising accuracy for in situ visualization of live US images. For tracking, no additional hardware nor modifications to HoloLens 2 are required making it a cheap and easy-to-use approach. Moreover, a minimal latency of displayed images enables a real-time perception for the sonographer.

Real-time integration between Microsoft HoloLens 2 and 3D Slicer with demonstration in pedicle screw placement planning

“…Second, we paid closer attention to the latency between the reslicing of the CT volume in 3D Slicer and its display in the glasses ( image reception ). This second analysis, which aligns with [ 14 ], allowed us to compare our system with the existing research in the field. This experiment was repeated 30 times for each time range.…”

“…To ensure effective communication across platforms, it is crucial that the exchange of information is seamless and allows for real-time perception. In line with the approach taken by [ 14 ], we measured latency by counting the number of frames it took from a change in one of the platforms to be reflected in the other one in a slow-motion video. The video was captured at a rate of 120 Hz through the glasses, where both the holographic information and the 3D Slicer screen containing the analogous models were always visible.…”

“…Our approach demonstrates the potential of AR to improve surgical planning by providing an immersive and interactive visualization of medical image data. Our project is based on the work presented in [ 14 ]. They displayed ultrasound (US) images in Microsoft HoloLens 2 utilizing the PLUS toolkit as the communicator between the AR glasses and a US probe.…”

Augmenting Image-Guided Procedures through In Situ Visualization of 3D Ultrasound via a Head-Mounted Display

“…All of these factors were likely an unwanted distraction throughout the task. Investigating other probe tracking approaches, e.g., using retroreflective spheres, as described in [ 11 ], would be an important step to further develop this system.…”

“…To minimize technical complexity, we preferred an inside-out tracking approach compatible with the HRM by using the built-in QR code tracking [ 10 ]. As the HRM does not provide access to the sensor data, we could not utilize a previously published tracking approach using retroreflective spheres [ 11 ]. For US data acquisition, we used an EPIQ 7 station (Philips Medical Systems International B.V., Best, The Netherlands) that allows real-time streaming of image data to the computer via a proprietary network protocol provided by Philips.…”