In recent years, infrared thermal polarization imaging technology has emerged as a new method of infrared detection. Its main objective is to detect, identify, and track the differences in infrared polarization characteristics between targets and scenes. Compared to ordinary infrared imaging systems, this technology offers significant advantages for detecting and identifying man-made targets in complex natural backgrounds. Specifically, the division of focal plane (DoFP) polarimeter is currently being extensively researched due to its compact structure, small volume, and real-time imaging capabilities. However, the focal plane array (FPA) of the DoFP polarimeter exhibits varying response gains and bias parameters, while the micro-polarizer array on its surface suffers manufacturing defects, resulting in significant nonuniformity errors that adversely affect resolution accuracy. This paper analyzes the mechanism behind the generation of non-uniformity in the DoFP polarization imaging system and establishes a relationship between the output grayscale of the system and incident irradiance. Additionally, through the use of a mid-infrared polarization radiation source, the proposed method conducts irradiation calibration and polarization parameter calibration to rectify the non-uniformity of the detection image of the system. Based on experimental results, the calibration model significantly enhances the accuracy of polarization detection information for scene targets.