To solve the problem of large measurement errors caused by vibrations of the surrounding environment, an error reduction method of a binocular measurement system is proposed. A light pen with a V-8 layout is designed to estimate the pose, and coded markers are used to establish a dynamic reference coordinate system to reduce the measurement error mainly caused by vibrations. The experimental results indicate that the light pen is highly accurate in pose estimation, and a root mean square error of the standard blocks and standard balls can be achieved within 0.02 mm when the measurement distance is 1.5 m. By measuring the straightness error of an ABB robot’s manipulator in a comparative experiment, the effective establishment of the dynamic reference coordinate system is preliminarily proven. The experiments are then conducted under vibrating conditions to calculate the standard deviations of the light pen probe center on the X-, Y-, and Z-axes with different quantities (3 and 4) of coded markers, which verify the high accuracy and real-time requirement of the method with a standard deviation below 0.040 mm and a processing speed above 45 frames per second when the measurement distance is 2 m.