With the increasing demands for long-endurance and high-accuracy inertial navigation system (INS), gravity disturbance has been identified as one of the major error sources with decisive effects on the performance of INS. To address the problem, this paper proposes an autonomous and high-accuracy gravity disturbance compensation scheme for rotary INS (RINS). The high-accuracy velocity measured by the laser Doppler velocimeter is fused with the angular velocity measured by the gyroscope to obtain navigation parameters, such as velocity, position and attitude, that are not affected by gravity disturbance. The navigation parameters independent of gravity disturbance are matched with the gravity disturbance-related navigation parameters output by RINS, and a measurement model containing gravity disturbance information is obtained. Besides, the intrinsic coupling relationship between gravity disturbance, gravity disturbance rate and gravity disturbance gradient is revealed, and a state-space model is established to accurately reflect the time-varying characteristics of gravity disturbance. Furthermore, the gravity disturbance is estimated and compensated in real-time through the optimal estimation algorithm. The results of vehicle experiments indicate that the gravity disturbance estimation precision of the proposed scheme is better than 2.15 mGal (1σ), and its horizontal position accuracy is better than 50 m at a driving distance of 80 km.