The precise matching of the optical system and the detector has attracted more and more attention to satisfy requirements of future infrared (IR) imaging systems, both for miniaturization and longdistance detection. Considering the trade-off between the spatial resolution and the sensitivity, this paper proposes a general optimization method, which can optimize parameters of the IR imaging system automatically by modifying λF/d. Firstly, on the basis of the trade-off between the spatial transfer characteristics and the spatial sampling characteristics, we establish a more accurate minimum resolvable temperature difference (MRTD) model, and analyze the coupling relationship between the optical system and the detector under the optimum MRTD. Secondly, we analyze the relationship between parameters of the IR imaging system and MRTD model, and propose a trade-off optimization approach of the IR imaging system. Finally, this paper verifies the rationality of the optimization method by optimizing CA-295 system. The trade-off optimization method proposed in this paper can not only satisfy the design requirement, but also reduce the size, weight, power consumption and cost (SWaP$) of IR imaging systems.