The human-machine integrated coordinate measurement is a promising coordinate measurement method with high flexibility and efficiency for the complex working environments. The cameras installed on the head-mounted measurement device (HMMD) achieves accurate global positioning by observing the uncoded LED landmarks, and then combines with the local measuring to obtain 3D coordinates. However, limited by the frame rate of the camera, the fast movements of the operator's head may cause landmark misidentification and visual positioning failure. In order to improve the robustness, a visual-inertial positioning method is proposed in this paper. An inertial measurement unit (IMU) is added to compensate for the deficiency of the visual positioning and enhance the dynamic performance. An adaptive extended Kalman filter (EKF), which adjusts the measurement noise covariance matrix based on the visual positioning uncertainty, is established to obtain the optimal state estimation. And an efficient initialization procedure is presented to implement the initial registration of uncoded landmarks based on the Normal Distribution Transform (NDT) algorithm and to determine the initial state of the IMU. Furthermore, the residual chi-square test is employed to detect false pose estimate in real time and to avoid positioning failure. The experiments demonstrate that the proposed method has high static positioning accuracy (0.681 mm) and high dynamic positioning robustness. The adaptive EKF realizes reliable landmark identification under fast movements and provides a higher accuracy than the common EKF.