The purpose of this study is to implement speckle reduced three-dimensional (3-D) holographic display by single phase-only spatial light modulator (SLM). The complex amplitude of hologram is transformed to pure phase value based on double-phase method. To suppress noises and higher order diffractions, we introduced a 4-f system with a filter at the frequency plane. A blazing grating is proposed to separate the complex amplitude on the frequency plane. Due to the complex modulation, the speckle noise is reduced. Both computer simulation and optical experiment have been conducted to verify the effectiveness of the method. The results indicate that this method can effectively reduce the speckle in the reconstruction in 3-D holographic display. Furthermore, the method is free of iteration which allows improving the image quality and the calculation speed at the same time.