Accurate polarization information acquisition is the key technique of the integrated inertial navigation system (INS)/polarization sensor (PS) navigation system. To solve the problem of poor performance of polarization sensors that real-time error calibration by using a rotary table, we propose a fast calibration method that polarization sensor combined with inertial navigation systems. Firstly, we construct the calibration model of polarization sensor. A tightly coupled INS/PS calibration model is proposed, specifically, the polarization sensor errors, which are the intensity gain coefficient, the degree of polarization coefficient, and the installation angle error of a polarizer, are augmented into the state of the integrated navigation systems. By introducing the error term of polarization sensor into the state of the integrated navigation systems, the error estimation can be carried out and the speed of error calibration can be improved. Secondly, considering the nonlinearity of error calibration model, we use the Unscented Kalman Filter (UKF) to calibrate the error. On this basis, we make use of the observability analysis to verify its validity of the proposed model in a dynamic scene. Finally, real-world experiments are conducted on the self-developed multi-channel polarization sensor. The experimental results show that the calibration method can be done quickly without the use of a rotary table.