High-fidelity pose estimation for real-time extended reality (XR) visualization for cardiac catheterization

Abstract

Extended reality (XR) technologies are emerging as promising platforms for medical training and procedural guidance, particularly in complex cardiac interventions. This paper presents a high-fidelity methodology to perform real-time 3D catheter tracking and visualization during simulated cardiac interventions. A custom 3D-printed setup with mounted cameras enables biplane video capture of a catheter. A computer vision algorithm processes the biplane images in real-time to reconstruct the 3D catheter trajectory represented by any designated number of points along its length. This method accurately localizes the catheter tip within 1 mm and can reconstruct any arbitrary catheter configuration. The tracked catheter data is integrated into an interactive Unity-based scene rendered on the Meta Quest 3 headset. The visualization seamlessly combines a reconstructed 3D patient-specific heart model with the dynamically tracked catheter, creating an immersive extended reality training environment. Our experimental study, involving six participants, demonstrated that the 3D visualization provided by the proposed XR system significantly outperformed 2D visualization in terms of speed and user experience. This suggests that the XR system has the potential to enhance catheterization training by improving spatial comprehension and procedural skills. The proposed system demonstrates the potential of XR technologies to transform percutaneous cardiac interventions through improved visualization and interactivity. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-024-76384-z.