A Method for Evaluating the Capacity of Generative Adversarial Networks to Reproduce High-order Spatial Context

Abstract

Generative adversarial networks (GANs) are generative models with the potential to revolutionize biomedical imaging. The overarching problem with GANs in clinical applications is a lack of adequate or automatic means of assessing the diagnostic quality of images generated by GANs. We demonstrate several statistical tests of images output by two popular GAN architectures. We designed several stochastic object models (SOMs) of distinct features that can be recovered after generation by a trained GAN. Several of these features are high-order, algorithmic pixel-arrangement rules which are not readily expressed in covariance matrices. We designed and validated statistical classifiers to detect the known arrangement rules, and tested the rates at which the GANs correctly reproduced the rules under a variety of training scenarios. We found that ensembles of generated images can appear accurate visually, and correspond to low Fr èchet Inception Distance, while not exhibiting the known spatial arrangements. Additionally, GANs trained on a spectrum of distinct spatial orders did not respect the order prevalence given in the training data. The main conclusion is that while low-order ensemble statistics are largely correct, there are numerous quantifiable errors per image that plausibly can affect subsequent use of the GAN-generated images.

Assessing the ability of generative adversarial networks to learn canonical medical image statistics

“…Although the FID score has shown excellent agreement with subjective visual assessments by humans [28], it is agnostic to the downstream task a medical image GAN may be used for. Additionally, it is an ensemble statistic, and hence could be blind to specific errors in high-order statistics of individual images [29].…”

“…Furthermore, the above mentioned measures are ensemble measures. It has been shown that individual samples drawn from the GAN may contain impactful errors despite giving satisfactory ensemble measures [29]. Lastly, medical image distributions typically consist of multiple classes or modes, and it has been shown that may produce critical errors while producing images from a mode that is rarely seen during training [18].…”

Mining the manifolds of deep generative models for multiple data-consistent solutions of ill-posed tomographic imaging problems

“…The StyleGAN must be sufficiently welltrained and should accurately represent the to-be-imaged object distribution. This can be challenging in diagnostic imaging applications, where the objects can contain varying pathologies that may not be fully represented in the StyleGAN training data [43], [44]. As such, the representation error of the Style-GAN should be acknowledged when employing PULSE++.…”

Evaluating Procedures for Establishing Generative Adversarial Network-based Stochastic Image Models in Medical Imaging

“…Despite the apparent realism and diversity in images, the extent to which a GAN learns image statistics is still an ongoing topic of research. [17][18][19] In particular, image GANs may produce images that appear visually realistic, but possess ensemble statistics that are incorrect, 19 which could be detrimental to downstream applications. 20 The Frechet inception distance (FID) score, inception score (IS) and average log-likelihood are a few among a long list of metrics that have been proposed to evaluate GANs.…”