The stripe fixed pattern noise (FPN) of infrared images significantly corrupts image quality, so that most infrared imaging systems suffer from the degradation of visibility and detectability during operation. Therefore, the FPN de-striping method, which eliminates stripe patterns without substantial loss of image information, remains a core technology in the field of infrared image processing. In this paper, we propose the dual-branch structure based FPN de-striping deep convolutional neural network (DBS-DCN) to effectively extract structural features of FPN and preserve the image details in a single infrared image. In addition, we have established the parametric FPN model through the diagnostic experiments of infrared images based on the physical principle of an infrared detector and its signal response. We have optimized each parameter of the FPN model using measured data, which acquired on a wide range of detector temperatures. Further, we generate the training data using our FPN model to ensure stable learning performance against various stripe patterns. We performed comparative experiments with state-of-the-art methods using artificially corrupted infrared images and real corrupted infrared data, and our proposed method achieved outstanding de-striping results in both qualitative and quantitative evaluation compared to existing methods.