HoloYolo: A proof‐of‐concept study for marker‐less surgical navigation of spinal rod implants with augmented reality and on‐device machine learning

Abstract

Background Existing surgical navigation approaches of the rod bending procedure in spinal fusion rely on optical tracking systems that determine the location of placed pedicle screws using a hand‐held marker. Methods We propose a novel, marker‐less surgical navigation proof‐of‐concept to bending rod implants. Our method combines augmented reality with on‐device machine learning to generate and display a virtual template of the optimal rod shape without touching the instrumented anatomy. Performance was evaluated on lumbosacral spine phantoms against a pointer‐based navigation benchmark approach and ground truth data obtained from computed tomography. Results Our method achieved a mean error of 1.83 ± 1.10 mm compared to 1.87 ± 1.31 mm measured in the marker‐based approach, while only requiring 21.33 ± 8.80 s as opposed to 36.65 ± 7.49 s attained by the pointer‐based method. Conclusion Our results suggests that the combination of augmented reality and machine learning has the potential to replace conventional pointer‐based navigation in the future.

Augmented reality in the operating room: a clinical feasibility study

“…Automated registration of the patient’s anatomy is currently the focus of various projects in augmented reality research [ 16 , 17 , 28 , 29 ]. Intraoperative manual surface digitization or machine learning based object detection offer the possibility to establish a correspondence between preoperatively acquired image data and intraoperative anatomy without further requiring intraoperative imaging [ 29 , 30 ]. Further, the quality of alignment of the three-dimensional objects to the real world not only relies on the accuracy of the registration but also on the calibration of the optical see-through head-mounted display.…”

The Click-On gamma probe, a second-generation tethered robotic gamma probe that improves dexterity and surgical decision-making

“…Alternatively, different groups have been promoting marker-less tracking of surgical instruments [39,40]. Unfortunately, for the latter, there still are many challenges for the realization of automated instrumentation segmentation without the use of markers: robustness of segmentation, occlusion, lighting condition, and depth estimation (the reported tracking accuracy also is in the 0.25-to 5-mm range) [41][42][43]. It is, however, expected that for most video-based tracking methods (including marker and marker-less approaches), further improvements will follow using rapidly evolving machinelearning algorithms [39,44].…”

On-Device Object Detection for More Efficient and Privacy-Compliant Visual Perception in Context-Aware Systems

“…There exists a prevalence in that regard, of widespread and cross-domain AmI applications on mobile or embedded devices, such as face detection [48,58,[60][61][62] (biometric security, surveillance), and vehicle and pedestrian detection (security, surveillance, autonomous vehicles, smart cities) both in cars [56,64,65] and unmanned aerial vehicles (UAV) [49,[52][53][54][55]57,59]. [73,74,88,97,110] and smart cities [72,100,101,108], all of them, scenarios where constant and real-time object detection is necessary for enabling context-awareness on end devices. While further information on each of those domains will be incorporated into the discussion in successive paragraphs to draw a clearer picture, it should be noted first that additional application areas, albeit almost residually with only one or two related works identified, have emerged in the analysis: (i) robotics [81,94], a domain where vision represents one of the most important communication channels with the environment, and where object detection has traditionally shown to be critical for the perception, modeling, planning, and understanding of unknown terrains [94]; (ii) defense, where object detection constitutes a major factor for controlling UAVs [84] and detecting ships in radar images [86]; (iii) smart logistics, with two distinct but equally representative examples of the use of sensing technologies, one on embedded platforms (in situ detection and recognition of ships for more efficient port management) [83], and the second one on mobile devices (barcode detection) [99] and finally, (iv) human emotion recognition based on facial expression detection, as reported in [71].…”

“…Finally, to conclude this first analysis focused on the current landscape of objectdetection-based AmI applications for low-power devices, we add to the discussion the two application domains not covered yet from the group of five with greater representation in the study: healthcare [73,74,88,97,110] and the so-called smart cities [72,100,101,108]. In regard to healthcare, on-device detection techniques are shown to be effective in extending healthcare spaces beyond the traditional scenario of closed clinical environments, bringing the capabilities of (i) disease diagnosis [73,74], (ii) wound or injury zone delimitation [97] and (iii) patient monitoring and support [88], (available only in typically complex and expensive configurations until recently) to low-cost portable devices.…”

“…Such research has resulted in significantly more efficient detection frameworks, making them directly deployable on low-power and low-memory hardware platforms but, on the flip side, causing a dramatic accuracy degradation. Despite the latter, these novel models, e.g., Tiny YOLO [119,120] and SSDLite [118], are already considered standard benchmarking options when designing new lightweight detectors in AmI [50,58,64,69,74,84,89,92,93,96,97,[101][102][103][108][109][110].…”