To address issues with poor image quality, limited material selection, and hard aberration correction in dual-band infrared (3.7–4.8 and 7.7–9.5 µm) zoom systems, we propose incorporating a high diffraction efficiency double-layer diffractive optical element (DLDOE) into the dual-band zoom system. Based on the three-component movable continuous zoom Gaussian model, a cold dual-band infrared refractive-diffractive hybrid zoom optical system with a DLDOE and 20× zoom ratio is designed. The system features a simple structure with only eight lenses and two types of materials. Image quality was assessed at three focal lengths in both middle-wave infrared (MWIR) and long-wave infrared (LWIR) ranges, and the size of the spot before and after the DLDOE was inserted was compared. The results show that the DLDOE ensures excellent image quality of the zoom system. This zoom system is characterized by a high zoom ratio, excellent image quality, and a simple structure, offering advantages in both military and commercial applications.