Real-Time Augmented Reality for Ear Surgery

Abstract

Transtympanic procedures aim at accessing the middle ear structures through a puncture in the tympanic membrane. They require visualization of middle ear structures behind the eardrum. Up to now, this is provided by an oto endoscope. This work focused on implementing a real-time augmented reality based system for robotic-assisted transtympanic surgery. A preoperative computed tomography scan is combined with the surgical video of the tympanic membrane in order to visualize the ossciles and labyrinthine windows which are concealed behind the opaque tympanic membrane. The study was conducted on 5 artificial and 4 cadaveric temporal bones. Initially, a homography framework based on fiducials (6 stainless steel markers on the periphery of the tympanic membrane) was used to register a 3D reconstructed computed tomography image to the video images. Micro/endoscope movements were then tracked using Speeded-Up Robust Features. Simultaneously, a micro-surgical instrument (needle) in the frame was identified and tracked using a Kalman filter. Its 3D pose was also computed using a 3-collinear-point framework. An average initial registration accuracy of 0.21 mm was achieved with a slow propagation error during the 2-minute tracking. Similarly, a mean surgical instrument tip 3D pose estimation error of 0.33 mm was observed. This system is a crucial first step towards keyhole surgical approach to middle and inner ears.

Video-based augmented reality combining CT-scan and instrument position data to microscope view in middle ear surgery

“…The aim of this study was to develop and assess a real-time AR system combining CT-scan data and microscopic video of the ear canal together with visualization of the surgical instrument behind closed tympanic membrane. This article extends our previous studies on AR based transtympanic procedures 15,22 by (1) validating the system's tracking schemes in near-realistic and challenging scenarios, and (2) imitating an actual procedure (drug administration), and (3) assessing the work in real-time instead of employing a recorded video. These developments brought the system several steps closer to its application in the operating room.…”

“…The collinear markers were extracted from the image using colour thresholding followed by pruning. However, due to small focus range of the microscope, the extraction was not perfect, and the centre points were extracted using blob detection 15 . The tool entry point was then used to associate the marker centres to marker labels B, C and D where B is closest to the instrument tip A, and D is closest to the tool entry point.…”

“…By setting the focal length of the camera as the z coordinate of the image projection points (b, c and d) and measuring the physical distance between markers (AB, BC, CD), the position of the instrument tip could be estimated using Eq. (4), by fitting the physical geometry (3D) of the tool onto the projected lines Ob, Oc and Od, where O is the origin of the camera axis 15 .…”

“…Moreover, in most studies the set-up required a conventional tracking system (mechanical, optical or electromagnetic). Similarly, in cranial base domain, mutual information, contour-based and point-based registration methods such as ICP have been widely used 10,13,[15][16][17][18] . Clinically, 5-10 minutes registration time has been regarded as acceptable with submillimetric precision 19 .…”

“…In a previous work, the applicability and performance of different tracking processes (using both endoscope and surgical microscope) was assessed on both human cadaveric temporal bones and artificial temporal bones 15,22 . The aim of this study was to develop and assess a real-time AR system combining CT-scan data and microscopic video of the ear canal together with visualization of the surgical instrument behind closed tympanic membrane.…”