The recent advance of synthetic image generation and manipulation methods allows us to generate synthetic face images close to real images. On the other hand, the importance of identifying the synthetic face images increases more and more to protect personal privacy from those. Although some deep learning-based image forensic methods have been developed recently, it is still challenging to distinguish synthetic images generated by recent image generation and manipulation methods such as the deep fake, face2face, and face swap. To resolve this challenge, we propose a novel generative adversarial ensemble learning method. We train multiple discriminative and generative networks based on the adversarial learning. Compared to the conventional adversarial learning, our method is however more focused on improving the discrimination ability rather than image generation one. To this end, we improve the discriminabilty by ensembling outputs from different two discriminators. In addition, we train two generators in order to generate general and hard synthetic images. By ensemble learning of all the generators and discriminators, we improve the discriminators by using the generated synthetic face images, and improve the generators by passing the combined feedback of the discriminators. On the FaceForensics benchmark challenge, we thoroughly evaluate our methods by comparing the recent methods. We also provide the ablation study to prove the effectiveness and usefulness of our method. INDEX TERMS Digital image forensics, generative adversarial ensemble learning, deep learning, synthetic image detection, face image.

Generating realistic images with fine details are still challenging due to difficulties of training GANs and mode collapse. To resolve this problem, our main idea is that leveraging the knowledge of an image classification network, which is pre-trained by a large scale dataset (e.g. ImageNet), would improve a GAN. By using the gradient of the network (i.e. discriminator) with high discriminability during training, we can, therefore, guide the gradient of a generator gradually toward the real data region. However, excessive negative feedback of the powerful classifier often prevents a generator from producing diverse images. Based on the main idea, we design a GAN including the added discriminator and propose a novel energy function in order to transfer the pre-trained knowledge to a generator and control the feedback of the added discriminator. We also present an incremental learning method to prevent the density of the generator to be the low-entropy distribution when training our GAN with respect to the energy function. We incorporate our method to DCGAN and demonstrate the ability to enhance the image quality even in high resolution on several datasets compared to DCGAN. In addition, we compare our method with recent GANs for the diversity of generated samples on CIFAR-10 and STL-10 datasets and provide the extensive ablation studies to prove the benefits of our method. INDEX TERMS Generative adversarial network, image classification network, image generation, generative model, deep learning, convolutional neural network.