Recently, a real objects-based full-color holographic display system usually uses a DSLR camera array or depth camera to collect data, and then relies on a spatial light modulator to modulate the input light source for the reconstruction of the 3D scene from the real objects. The main challenges faced by the holographic 3D display were introduced, including limited generation speed and accuracy of the computer-generated holograms, the imperfect performance of the holographic display system. In this research, we generated more effective and accurate point cloud data by developing a 3D saliency detection model in the acquisition module. Object points categorized into depth girds with identical depth values in the red, green, and blue (RGB) channels. In each channel, the depth girds are segmented into M × N parts, and only the effective area of the depth grids will be calculated. Computer-generated holograms (CGHs) are generated from efficient depth grids by using Fast Fourier transform (FFT). Compared to the wave-front recording plane (WRP) and traditional PCG methods, the computational complexity is dramatically reduced. The feasibility of the proposed approaches is established through experiments.