In order to constantly and accurately recognize and track flying targets, using the infrared photoelectric detection system, while scientifically analyze the detection capability of the system, in complex background environments. This paper analyzes the measurement principle of the infrared photoelectric detection system, establishes a radiation energy calculation model of the background and the optical system and the target on the photoelectric detector surface. It further analyzes the influence of the photoelectric detector wavelength, the slit diaphragm and of the target effective radiation area, on the radiation energy. It obtains the contrast ratio calculation model between the target and the background, on the photoelectric detector surface, while it also derives the detection distance model of the system, combining the modulation transfer function of the system, to set up an improved detection distance calculation model, in complex background environments. Simulation and experimental analysis show that as background radiation brightness increases, in certain background environments, the detection distance of the system gradually reduces, while changing the slit diaphragm and photoelectric detector wavelength can improve the detection performance of the system. In complex background environments, changing the contrast ratio of photoelectric detector surface should be considered, to improve the detection capability of the system.