On Aliased Resizing and Surprising Subtleties in GAN Evaluation

Abstract

We investigate the sensitivity of the Fréchet Inception Distance (FID) score to inconsistent and often incorrect implementations across different image processing libraries. FID score is widely used to evaluate generative models, but each FID implementation uses a different low-level image processing process. Image resizing functions in commonlyused deep learning libraries often introduce aliasing artifacts. We observe that numerous subtle choices need to be made for FID calculation and a lack of consistencies in these choices can lead to vastly different FID scores. In particular, we show that the following choices are significant: ( 1) selecting what image resizing library to use, ( 2) choosing what interpolation kernel to use, ( 3) what encoding to use when representing images. We additionally outline numerous common pitfalls that should be avoided and provide recommendations for computing the FID score accurately. We provide an easy-to-use optimized implementation of our proposed recommendations in the accompanying code.

CG-NeRF: Conditional Generative Neural Radiance Fields

“…As reported in Table 2, to guarantee the most reliable performance of the previous methods, we evaluate the comparison results using a publicly available pre-trained model and its corresponding experiment setting. Because these visual quality metrics are highly sensitive to measurement tools showing incorrect implementations across different image processing libraries (Parmar, Zhang, and Zhu 2021), we evaluate all of the comparison methods using the same measurement tools. Based on the performances we measured, the proposed method shows better scores in terms of FID, precision, and recall compared to the existing is close to that of the real image in terms of the mean and standard deviation.…”

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

“…We additionally compare with StyleGAN2 [25] in the unconditional setting. We use Clean-FID [44] for benchmarking due to its reported benefits over previous implementations of FID [16]. 3 Results on MM-CelebA-HQ and MS-COCO are summarized in Tab.…”

Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN

“…Metrics. We apply the Fréchet Inception Distance (FID) [27] to measure the similarity between real and synthesized images, and perform human evaluation to quantitatively evaluate the synthesis quality of different methods. For the human evaluation, we design three questionnaires corresponding to the three used datasets.…”

“…4 We directly use the pre-trained model of these methods as their training procedure depends on the paired data of garment-person or person-person image pairs, which are unavailable in our dataset. When testing paired methods under the unpaired try-on setting, we extract the desired garment from the person image and regard it as the in-shop garment to meet the Table 1: The FID score [27] and human evaluation score among different methods under the unpaired setting on the DeepFashion dataset [21] and our UPT dataset. need of paired approaches.…”

“…The FID score[27] and human evaluation score among different methods under their corresponding test setting on the MPV dataset[6].…”