We put forward a novel hybrid iterative algorithm to improve the imaging quality of digital holography. An off-axis hologram is added to the iteration process via interference and inverse interference process and becomes part of the constraints. A frequency domain filter varying with the number of iterations is used to improve the competitive advantage of low frequency information in the early iterations, while retaining the high frequency information. In practical applications, an additional iterative process is used after averaging filtering to suppress the influence of the imperfect consistency between the reconstructed reference wave and the actual reference wave. Numerical simulations and experiments show that image reconstruction may be significantly improved compared to the conventional method.