The major challenge of current strapdown inertial navigation system/celestial navigation system (SINS/RCNS) is failing to calibrate all the error coefficients of gyro and star sensor in-flight accurately, which will lead to the attitude accuracy degradation. To address this question, this paper proposes an optimal calibration method based on observability analysis. Here, we derive the star sensor measurement model with respect to initial alignment errors, gyro errors, and star sensor installation errors. And the observability analysis is employed to explain the reason why all the error coefficients can be estimated effectively by three maneuvers merely. Finally, the optimal observation sequence is determined based on the fisher information matrix (FIM). The proposed optimal calibration method is evaluated by a representative suborbital flight vehicle trajectory, which represents significant improvements in estimation accuracy of error coefficients, and realizes the attitude accuracy enhancements.