Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Abstract

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with threedimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient's anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient's anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals. Video LinkThe video component of this article can be found at https://www.jove.com/video/60618/ 2 . These methods have been considered somewhat limited when an accurate registration is required 19 . To overcome these limitations, this work provides tools to perform accurate and simple patient-to-image registration in AR procedures by combining AR technology and 3DP.The protocol is generic and can be applied to any medical imaging modality or patient. As an example, a real clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. The first step describes how to easily segment the affected anatomy from Journal of Visualized Experiments www.jove.com Copyright © 2020 Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License January 2020 | 155 | e60618 | Page 2 of 10 computed tomography (CT) medical images to generate 3D virtual models. Afterward, positioning of the 3D models is performed, then the required tools and models are 3D-printed. Finally, the desired AR app is deployed. This app allows for the visualization of patient 3D models overlaid on a smartphone camera in real-time.3. Click on the Show 3D button to view a 3D representation of the segmentation.3. Export the segmentation in a 3D model file format by going to the Segmentations module in 3D Slicer.1. Go to Export/import models and labelmaps. Select Export in the operation section and Models in the output type section. Click Export to finish and create the 3D model from the segmented area. 2. Selec...

Conceptual evolution of 3D printing in orthopedic surgery and traumatology: from “do it yourself” to “point of care manufacturing”

“…The products requested were 3D printed biomodels in 87.32% of cases and surgical positioning guides in 10.75%. The remaining cases were patient-specific instruments for surgical interventions, in which 3D printing hybridization with surgical navigation or augmented reality provided added value [18,19] (Fig. 2).…”

Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation

“…Each guide included holes (Ø 5 mm) to attach it to the bone using screws, and a holder for the AR tracking marker that will allow automatic patient-to-image registration. The registration transformation was obtained with a previously developed 3D Slicer module [ 35 ].…”

“…The smartphone application, ARHealth , was developed in Unity (version 2019.3), using C# programming language, and is compatible with Android and iOS devices. The app uses Vuforia SDK (Parametric Technology Corporation Inc., Boston, MA, USA) to identify the patterns of a cubic reference marker [ 35 ] in the smartphone camera field of view (FOV) and project the virtual models overlaid onto the real-world image. These models will be correctly registered with the patient since their relative coordinates are computed and stored using a previously developed 3D Slicer module [ 35 ], and then recovered by ARHealth .…”

“…The app uses Vuforia SDK (Parametric Technology Corporation Inc., Boston, MA, USA) to identify the patterns of a cubic reference marker [ 35 ] in the smartphone camera field of view (FOV) and project the virtual models overlaid onto the real-world image. These models will be correctly registered with the patient since their relative coordinates are computed and stored using a previously developed 3D Slicer module [ 35 ], and then recovered by ARHealth . We designed the cubic marker (30 × 30 × 30 mm 3 ) to contain unique patterns in black and white on each face ( Figure 3 ).…”

“…Previous studies have shown some initial results of this approach: Witowski et al designed a workflow that reduced anesthetic time, morbidity and postoperative complications [ 33 ]; our group implemented an AR system that enabled automatic registration with a 3D-printed patient-specific surgical guide [ 34 ]. The integration of 3D printing and AR could be useful not only during the surgical intervention, but also in all surgical workflow stages [ 35 ]. In this study, we propose a new surgical framework including both technologies through the treatment process in orthopedics oncology.…”