A circular grating angle encoder is a key component in the dynamic torque calibration system. To improve the accuracy of an angle measurement, in this paper, the source of the angle measurement error of the circular grating is analyzed; an eccentricity error model and an inclination error model are proposed, respectively; further, these two models are combined to establish a total error model. Through the simulation study with the models, the conditions, in which the eccentricity error or inclination error can be ignored, are discussed. The calibration and compensation methods of the angle measurement error are given, and a progressive error compensation function which integrates the first harmonic fitting and the second harmonic fitting is obtained. An experiment is performed to verify the proposed calibration and compensation methods. The peak-to-peak value of the compensated angle measurement error of the single reading head can be reduced by about 93.76%, which approximates to the error of the mean value of the double reading heads. The experimental results show that the error calibration and compensation method based on the proposed error model can effectively compensate the angle measurement error of the circular grating with a single reading head, and obtain a high-precision measurement angle.