Frequency domain characteristics and optimization of image generation for GANs

Abstract

In recent years, Generative Adversarial Networks (GANs) have demonstrated enormous promise in areas connected to image generation. As the model generation performance continues to improve and the generated images become more realistic, it is difficult to effectively distinguish between the real image and the generated image. Therefore, the problem of discriminating and optimizing the generated images (adversarial discrimination) has become necessary, and subsequent optimization plans are proposed based on the discrimination strategy. However, due to the nature of convolution, the two-dimensional power spectrum curve of the generated image is low overall; that is, compared with the real image, there is energy loss at each frequency (without other processing), and the curve drops rapidly and approaches zero, which is obviously different from the real image. In particular, the curve of the image generated by transposed convolution has a clear upward trend at the very high-frequency part, which is contrary to the characteristic of the real image, which is that the energy decreases with increasing frequency. Based on the discussion of the characteristics and inducements of the two-dimensional power spectrum curve of the generated image, we present a discrimination approach based on curve warping at high frequency and energy loss to improve the discrimination capacity of the generated image and realize the effective discrimination between the real image and the generated image. Based on this, we present the power spectrum loss function to improve the upward warping characteristics of the very high-frequency part of the two-dimensional power spectrum curve without degrading the quality of the generated image and the high-frequency feature loss function to improve the quality of the generated image. The value and efficiency of the proposed discrimination approach in this study are demonstrated on multiple GANs models, including WGAN, WGAN-GP, and SAGAN, with the dataset celeba, and the GANs model with encoder-decoder as the generator with the dataset CelebA-HQ. The two loss functions proposed are also demonstrated on multiple GANs models, including WGAN, WGAN-GP, and SAGAN with the dataset FFHQ. After adding the high-frequency feature loss, the FID decreases by 5.97, 5.15, and 6.56, respectively. After adding the power spectrum loss, the above models can improve the upward warping characteristics of the two-dimensional power spectrum curve in the very high-frequency part of the generated image to a certain extent. The FID decreases by 17.4, 11.55 and 12.27 when the weight is fixed, and 12.66, 8.15 and 4.46 when the weight is variable, respectively.