This study demonstrates that diffusion doping of ZGP single crystals with impurity atoms (Mg, Se, Sn, Pb) leads to a decrease in the specific conductivity of the samples. Consequently, this results in reduced absorption in the terahertz frequency range (150–1000 μm). It has been shown that doping ZGP samples with selenium (Se) and lead (Pb) atoms reduces absorption in the infrared region from 0.3–0.6 cm−1 to 0.06–0.09 cm−1. Doping with tin (Sn) leads to a decrease in absorption only in the wavelength region near 2.1 μm from 0.2 cm−1 to 0.05 cm−1. The proposed mechanism for the decrease in infrared absorption is a reduction in zinc vacancies due to doping with impurity atoms. This research lays the groundwork for a technology that produces ZGP crystals with minimal absorption within the 2–8 μm wavelength range, eliminating the need for fast electron beam irradiation technology. This advancement will facilitate the fabrication of ZGP crystals with arbitrary apertures.