To address the main section misalignment of rotating Risley prisms in a camera imaging system, a zero calibration method for a Risley prism main section is proposed according to the dispersion characteristics of the prisms–camera imaging. The principal component analysis method and the Radon transform method are combined to establish an effective mapping relationship between the prism main section positions and the prismatic dispersion characteristics using the camera dispersion image as references. The estimation of the dispersion direction was realized by referring to the motion blur estimation method and verified by experiment. As a result, the target-free self-calibration of the prism main sections in the camera coordinate system can be carried out under a prisms–camera intact assembly mode.