Rotary optical encoder is an important component in rotational inertial navigation system. It is used to form a closed loop motor control system and calculate the system attitude. The system performance will be affected by the encoders’ error. Besides the installation errors, the working temperature variants will also lead to encoders’ error. Therefore, the paper proposes a method to calibrate and compensate temperature resulted errors for rotary optical encoders. Firstly, an independent testing mechanism with position limitation and rotatable platform is designed and produced to verify the temperature influences on encoders. Then, the temperature error of rotary optical encoder used in rotational inertial navigation system is calculated by the gyroscope whose sensitive axis is parallel to the same motor axis. The method is verified by a self-researched single axis rotational inertial navigation system. According to the experimental results, the attitude accuracy is increased more than 36.7% compared with traditional method.