The spectral polarization measurement can obtain not only the spectral information of the target but also its polarization information, which can improve the detection and identification of the measured target. In the polarization spectrometer based on a liquid crystal variable retarder (LCVR) and acousto-optic tunable filter (AOTF), the LCVR is a core device for achieving fast and high-precision polarization detection. The AOTF is a new, to the best of our knowledge, filter device for spectral tuning. To reduce the sensitivity of an LCVR-based Stokes polarization spectrometer system to errors and Gaussian noise, and to maintain the advantage of fast electrical tuning of the system for spectral polarization detection, the phase retardation and azimuth angle of the polarization device LCVR is calculated and analyzed optimally under the minimum number of samples N=4 of the Stokes vector measurement method in this paper. The optimization algorithm considers the constraints, such as the number of types of LCVR phase retardation and the number of adjustments, and the azimuth and phase retardation to be optimized are searched for optimality step by step. The simulation results show that the number of adjustments of the phase retardation δ of LCVRs is only three times when four Stokes parameters are obtained. The LCVRs’ number of species is four kinds (2×2). The condition number of the optimized measurement matrix is 1.742, which converges to the ideal condition number, the optimal azimuth angle (θ1,θ2) is (18.9°, 41.9°), and the optimal phase retardation δ is (179.9°, 156.6°, 0.4°, 46.3°). Its corresponding tetrahedral volume is closer to the ideal value. The optimized system is less sensitive to errors and Gaussian noise.