In iterative pseudo-inverse ghost imaging (IPGI), how much the noise interference item of the current iteration approximates the real noise greatly depends on the clarity of initial image. In order to improve IPGI, we propose a method that introduces anisotropic diffusion to construct a more accurate noise interference term, where anisotropic diffusion adapts to both the image and the noise, so that it balances the tradeoff between noise removal and preservation of image details. In our algorithm, the anisotropic diffusion equation is used to denoise the result of each iteration, then the denoised image is used to construct the noise interference term for the next iteration. Compared to IPGI, our method has better performance in visual effects and imaging quality, as the image edges and details are better preserved according to the experimental results.