An Online Calibration Method for Microsoft HoloLens

Abstract

An optical see-through (OST) head-mounted display (HMD) is an appropriate option for an image-guided surgical navigation system (IGS) based on augmented reality (AR). However, the calibration of OST-HMDs has been challenging, since the augmented results can only be observed by the users wearing it. In this paper, we proposed an online calibration method based on the single-point active alignment method (SPAAM) for a HoloLens using a virtual tracking system. The transformation matrices between the world coordinates and the virtual holographic coordinates are calculated using a linear model and decomposed by the singular value decomposition (SVD) algorithm. The surgical workspace is analyzed, and there are nine sampling points for the offline calibration: eight at the vertices and one at the center of a cube of the workspace. 20 groups of data at each sampling point are collected. Five alignments should be implanted in the online calibration. The transformation matrices are compensated for the offset of every use of a HoloLens. To assess the accuracy of the calibration method, in accordance with the RANSAC algorithm, eight groups of additional data at random non-sampling locations are collected. Three different users that are familiar with the calibration procedures performed the calibration. The average distance error of our calibration was below 6 mm, and the rotation error was up to 5 •. It is a user-friendly, simple method for the online calibration of OST-HMDs. INDEX TERMS Optical see-through, augmented reality, HMD calibration.

Alignment-Free Offline Calibration of Commercial Optical See-Through Head-Mounted Displays With Simplified Procedures

“…The higher error and variance of the object-based method are expected since the camera-based calibration is tracking-independent whereas the object-based method directly relies on the tracked target pose. As the head-anchored tracking system is used in our experiment, the tracked target position can drift due to the "accumulative errors among sensors" [11]. Also, the asynchronism between the self-tracking and display refresh could lead to a jiggly display.…”

“…In VST systems, the environment is recorded by one or two cameras and displayed to the user: the view of the real world is mediated by the camera(s). By contrast, in OST displays the "video stream" directly comes from the user's eye [11], and therefore it is not possible to use standard image processing techniques to align virtual contents with the scene [12]- [14]. Effective and efficient OST calibration represents an open research problem.…”

“…The performance was tested on HoloLens and Moverio BT-300 with both head-anchored and world-anchored tracking. Guo et al implemented an online SPAAM-based calibration method for HoloLens [11]. The display was first calibrated in the entire workspace with a time-consuming offline calibration by 9×20 alignment tasks.…”

“…The accuracy of OST-HMD calibration cannot be objectively evaluated as we do not have direct access to the augmented image formed on the user's retina [11]. For alignment-based calibration methods, the tracked target can be re-projected to the display using the calibrated projection matrix and the reprojected pixels are then compared to "ground-truth" pixels that are manually annotated on the virtual display plane by users [40], [41] or by dividing the calibration data sets into training and evaluation blocks [19].…”

Review of Microsoft HoloLens Applications over the Past Five Years

“…For instance, a study on effectively controlling the HoloLens, using the gaze, gesture and voice control functions provided by the device, and studies on methods for utilizing this device in various industrial fields have been conducted [7][8][9][10][11][12][13][14][15][16][17]. Research on methods of visualizing the scanned image or data of an actual object as 3D contents [18][19][20] and correcting them [21,22] have also been conducted. Moreover, some studies have evaluated the performance and utility of the HoloLens, while others have developed additional toolkits or hardware to improve the performance of the HoloLens [2,[23][24][25][26].…”