The rotational inertial navigation system (RINS) could use rotational alignment to improve azimuth alignment accuracy significantly. However, due to the existence of magnetic drifts of fiber optic gyros, under the comprehensive influence of the outside world, there will be residual drifts that are unable to be averaged out along the body frame or the geographic frame, which will have a significant impact on the azimuth alignment accuracy. An innovative and effective calibration compensation strategy is proposed by analyzing the generation mechanism of the residual drift and its influence on alignment accuracy, which can significantly improve the azimuth alignment accuracy. With the aid of a high-precision total station, the experimental results of the initial alignment in four directions show that the accuracy of azimuth alignment is improved from 16.5'' to 10.3'' by compensating the residual drifts, which has significant theoretical significance and practical value of engineering application for improving the overall performance of inertial navigation.