The device errors of a rotating-wave-plate-based polarization measurement system can be mainly categorized as the angle error of the polarization prism, the fast axis angle error of wave-plate and the retardation error. By applying Fourier analysis to solve Stokes vectors, we obtain the formulas to calculate the three errors mentioned above, using linear 0° and 45° polarized light to illuminate the system for the error solving. We analyze the measurement errors of the degree of polarization, the polarization purity and the intensity of polarization state, under certain simulation conditions. The results show that after the calibration, the measurement accuracy is improved. Finally, we analyze the effects of the angle error of the linear polarized light used for calibration to the measurement of the three parameters.