The error caused by measurement itself directly affects the accuracy of models. The error caused by the failure to follow the Bryan principle in the process of measuring the volumetric error of the machine tool makes the geometric accuracy of the machine tool after compensation less than the ideal effect. In this paper, it is proposed that the measurement error will be caused by the non coincidence of the measurement point and the tool position point in the measurement of linear axis error. According to Bryan principle, the relationship between the center point of the reflective mirror of the laser interferometer and the tool position point is analyzed. The straightness error produced in the measurement process is introduced into the volumetric error of the machine tool, and the geometric model containing the measurement error is established to improve the modeling accuracy of the machine tool. To prove the effectiveness of compensating measurement errors, an experiment has been conducted on a three-axis vertical machining center. The experimental results show that the geometric accuracy of the machine tool is improved by 23% after the measurement error is compensated.