Traditional methods for measuring geometric errors in machine tools, including interferometry and Double Ball Bar (DBB), are known to be expensive and time-intensive. Consequently, a non-contact calibration system called the “Laser R-test” has been developed. This innovative system is designed to measure both position-independent geometric errors (PIGEs) and position-dependent geometric errors (PDGEs) efficiently. Since its development in 2000, this tool has been instrumental in analyzing eccentricity errors, angular position errors, and simultaneous trajectory errors. Through extensive research, it has been determined that the total error in a five-axis machine tool can be controlled to below 40 µm after compensating for eccentricity parameters and angular position errors. However, reducing this error to below ± 10 µm is challenging, primarily due to wobble errors in the orientation of the rotary axis without compensating. In this study, a new methodology based on Laser R-test and Rodrigues’ rotation formula has been developed to establish a PIGE model of rotary axis. Based on the methodology, the 8 PIGEs can be analyzed by measuring 5 coordinate positions. The compensation of 8 PIGEs in the rotary axis is completed within 30 min using the inspection path. Compatibility with ISO-10791–6 standards for BK1, BK2, and BK4 path tests is confirmed, validating the compensation effects. A precision of below ± 10 µm is achieved, with inspection time reduced by over 50%. This system can complete multiple errors by simply using the different paths. This greatly reduces the setup time for future users, enhancing its commercial applicability.