Current development in a deep neural network (DNN) has given an opportunity to a novel framework for the reconstruction of a holographic image and a phase recovery method with real-time performance. There are many deep learning-based techniques that have been proposed for the holographic image reconstruction, but these deep learning-based methods can still lack in performance, time complexity, accuracy, and real-time performance. Due to iterative calculation, the generation of a CGH requires a long computation time. A novel deep generative adversarial network holography (GAN-Holo) framework is proposed for hologram reconstruction. This novel framework consists of two phases. In phase one, we used the Fresnel-based method to make the dataset. In the second phase, we trained the raw input image and holographic label image data from phase one acquired images. Our method has the capability of the noniterative process of computer-generated holograms (CGHs). The experimental results have demonstrated that the proposed method outperforms the existing methods.