In order to achieve high sensitivity, accuracy and fast measurement of ellipsometry system, a phase modulation ellipsometry system based on Photoelastic Modulator (PEM) is adopted. At present, the phase delay of the Photoelastic Modulator is generally obtained by using the digital Lock-in Amplifier to extract the harmonic component and the DC component. The high frequency signal is discarded and the static phase delay of the Photoelastic Modulator is set as constant in the calculation process using the digital Lock-in Amplifier. However, the fluctuation of system parameters such as temperature, incident Angle and input signal will cause the decrease of measurement accuracy. In order to avoid the measurement error caused by traditional methods, an optimization method is proposed to calibrate the ellipsometry system with phase modulation. The nonlinear least square method is used to transform the ellipsometry measurement system into an optimization model, and the best match between the estimated value and the observed value is found. In order to verify the proposed optimization calibration method, the digital Lock-in Amplifier and the optimization method were used to scale the measurement system respectively, and the phase delay under the two methods were obtained respectively. The calculated deviations of the calibration of the two methods were 0.1805 and 0.141rad respectively, and the fluctuation of 1h calibration results was less than 1%. The results show that this method can reduce the influence of system parameter error on the measurement accuracy.