Augmenting Image-Guided Procedures through In Situ Visualization of 3D Ultrasound via a Head-Mounted Display

Abstract

Medical ultrasound (US) is a commonly used modality for image-guided procedures. Recent research systems providing an in situ visualization of 2D US images via an augmented reality (AR) head-mounted display (HMD) were shown to be advantageous over conventional imaging through reduced task completion times and improved accuracy. In this work, we continue in the direction of recent developments by describing the first AR HMD application visualizing real-time volumetric (3D) US in situ for guiding vascular punctures. We evaluated the application on a technical level as well as in a mixed-methods user study with a qualitative prestudy and a quantitative main study, simulating a vascular puncture. Participants completed the puncture task significantly faster when using 3D US AR mode compared to 2D US AR, with a decrease of 28.4% in time. However, no significant differences were observed regarding the success rate of vascular puncture (2D US AR—50% vs. 3D US AR—72%). On the technical side, the system offers a low latency of 49.90 ± 12.92 ms and a satisfactory frame rate of 60 Hz. Our work shows the feasibility of a system that visualizes real-time 3D US data via an AR HMD, and our experiments show, furthermore, that this may offer additional benefits in US-guided tasks (i.e., reduced task completion time) over 2D US images viewed in AR by offering a vividly volumetric visualization.

Real-Time Simulation of Wave Phenomena in Lung Ultrasound Imaging

“…In the following step, the Unity framework was used to create a simulation platform for the Lung Ultrasound Simulator. The Unity framework is widely utilized for developing 2D and 3D games, and it has also found extensive application in the medical field [38][39][40][41]. Moreover, it serves as a solid foundation for creating web-based 3D applications, offering a versatile platform for delivering immersive experiences over the Internet.…”

Mixed reality guided root canal therapy

“…Considering the requirements for freedom of movement and accuracy in medical AR applications, two main types of systems have been developed: on‐device tracking systems and hybrid AR systems. On‐device tracking systems [ 36 , 37 , 38 , 39 ], such as HoloLens 2 [ 40 ], utilize the power of the device itself to capture real‐world scenes, as well as to track and register virtual objects in real‐time without the need for external tracking systems. These on‐device tracking AR systems provide a more streamlined and efficient user experience with more manoeuverability.…”

“…Specifically, to precisely align virtual models with real‐world objects [ 42 ], and ensure seamless integration between the virtual and physical domains, three distinct approaches to pose computation and registration have been described (Table 1 ). The first approach is marker‐based [ 17 , 22 , 36 , 43 , 44 , 45 , 46 ], where identifiable fiducial markers are attached to the physical objects to facilitate accurate tracking. These markers serve as reference points, enabling the system to precisely determine the position and orientation of the objects.…”

Visualization modality for augmented reality guidance of in‐depth tumour enucleation procedures

“…With this guidance modality, 80% of the users were able to biopsy a 5 mm diameter target lesion. Further progress in this direction was achieved by reference [ 11 ] in 2023, which developed the first application allowing in situ visualisation of real‐time volumetric US acquisition for vascular puncture guidance. The authors compared the combination of 3D US with AR versus 2D US with AR to determine whether the volumetric visualisation could indeed offer additional benefits in procedure outcomes.…”