In holographic three-dimensional (3D) display, phaseonly spatial light modulators (SLMs) can only accept pure phase modulation signals, often leading to amplitude distortion in the reconstructed image and inaccurate object reconstruction. This paper presents a new modulating phase encoding method with amplitude compensation. Based on the imaging characteristics of phase digital holograms, the object light amplitude variation is extended to the nonlinear region of the first-order Bessel function to maximize the intensity of the reconstructed 3D image. The amplitude distortion of the reconstructed image is effectively eliminated by pre-distortion treatment of the light amplitude. Theoretical simulations and optical experiments validate the presented method.