Development of an inside-out augmented reality technique for neurosurgical navigation

Abstract

OBJECTIVE With the advancement of 3D modeling techniques and visualization devices, augmented reality (AR)–based navigation (AR navigation) is being developed actively. The authors developed a pilot model of their newly developed inside-out tracking AR navigation system. METHODS The inside-out AR navigation technique was developed based on the visual inertial odometry (VIO) algorithm. The Quick Response (QR) marker was created and used for the image feature–detection algorithm. Inside-out AR navigation works through the steps of visualization device recognition, marker recognition, AR implementation, and registration within the running environment. A virtual 3D patient model for AR rendering and a 3D-printed patient model for validating registration accuracy were created. Inside-out tracking was used for the registration. The registration accuracy was validated by using intuitive, visualization, and quantitative methods for identifying coordinates by matching errors. Fine-tuning and opacity-adjustment functions were developed. RESULTS ARKit-based inside-out AR navigation was developed. The fiducial marker of the AR model and those of the 3D-printed patient model were correctly overlapped at all locations without errors. The tumor and anatomical structures of AR navigation and the tumors and structures placed in the intracranial space of the 3D-printed patient model precisely overlapped. The registration accuracy was quantified using coordinates, and the average moving errors of the x-axis and y-axis were 0.52 ± 0.35 and 0.05 ± 0.16 mm, respectively. The gradients from the x-axis and y-axis were 0.35° and 1.02°, respectively. Application of the fine-tuning and opacity-adjustment functions was proven by the videos. CONCLUSIONS The authors developed a novel inside-out tracking–based AR navigation system and validated its registration accuracy. This technical system could be applied in the novel navigation system for patient-specific neurosurgery.

Three-Dimensional Modeling and Extended Reality Simulations of the Cross-Sectional Anatomy of the Cerebrum, Cerebellum, and Brainstem

“…To our knowledge, this study is the first to physically section and digitalize an entire cadaveric brain to create an interactive 3D model in the 3 principal imaging planes: axial, sagittal, and coronal. Previous studies [10][11][12][13][14][15][16][17][18][19][20][21][22] with 3D renderings of the brain lack texture, realistic features, maneuverability, augmentation, and fine cross-sections through multiple planes. Similarly, available cross-sectional radiological images and anatomic photographs of the human brain provide only (For this specific study, our primary aim was to present 3-dimensional cross-sectional photogrammetric anatomy of human brain.…”

“…38,39 Previous studies have incorporated 3D technology for neurosurgery practice, training, and education. [10][11][12][13][14][15][16][17][18][19][20][21][22] However, most of these studies used artificial textures overlaid on reconstructed data from CT or MRI. 10,11,13,17,18 Others used realistic textures but lacked complete 360°maneuverability.…”

“…[10][11][12][13][14][15][16][17][18][19][20][21][22] However, most of these studies used artificial textures overlaid on reconstructed data from CT or MRI. 10,11,13,17,18 Others used realistic textures but lacked complete 360°maneuverability. 12,14,15,[19][20][21][22] By contrast, our study models combine realistic features with 360°movement.…”

Navigation of frameless fixation for gamma knife radiosurgery using fixed augmented reality

“…For cases when automatic registration using the QR marker could be misaligned, the correction function of fixed-AR was developed and could alter the location of the virtual 3D model of AR navigation based on the following principles 9 : the virtual space is shown on the device screen through the coordinates (local, world, and camera); the local coordinate is a part of the virtual 3D model itself. The world coordinate refers to the virtual space wherein the model is placed, while the camera coordinate refers to the coordinate system wherein the world coordinate is viewed from the standard reference of the camera.…”

“…Although inside-out tracking has less accuracy compared to outside-in tracking, installing the equipment for visualization and object detection for registration is low cost. We previously reported on an inside-out tracking-based AR-neuro-navigation system using ARKit® (Apple Inc.) based software 9 . We applied the inside-out tracking for radiosurgery by mounting it to a 4th generation iPad Pro (Apple Inc. San Francisco, USA) to test the feasibility to develop clinically usable inside-out tracking AR in gamma knife radiosurgery (GKRS), which utilizes devices in a fixed-stated, called fixed-AR.…”