This paper proposes a recognition methodology for key geometric errors using the feature extraction method and accuracy retentivity analysis and presents the approach of optimization compensation of the geometric error of a multiaxis machine tool. The universal kinematics relations of the multiaxis machine tool are first modelled mathematically based on screw theory. Then, the retentivity of geometric accuracy with respect to the geometric error is defined based on the mapping between the constitutive geometric errors and the time domain. The results show that the variation in the spatial error vector is nonlinear while considering the operation time of the machine tool and the position of the motion axes. Based on this aspect, key factors are extracted that simultaneously consider the correlation, similarity, and sensitivity of the geometric error terms, and the results reveal that the effect of the position-independent geometric errors (PIGEs) on the error vectors of the position and orientation is greater than that of the position-dependent geometric errors (PDGEs) of the linear and rotary axes. Then, the fruit fly optimization algorithm (FOA) is adopted to determine the compensation values through multiobjective tradeoffs between accuracy retentivity and fluctuation in the geometric errors. Finally, an experiment on a four-axis horizontal boring machine tool is used to validate the effectiveness of the proposed approach. The experimental results show that the variations in the precision of each test piece are lower than 25.0%, and the maximum variance in the detection indexes between the finished test pieces is 0.002 mm when the optimized parameters are used for error compensation. This method not only recognizes the key geometric errors but also compensates for the geometric error of the machine tool based on the accuracy retentivity analysis results. The results show that the proposed methodology can effectively enhance the machining accuracy.