Pros and cons of GAN evaluation measures

Borji, Ali

doi:10.1016/j.cviu.2018.10.009

Cited by 720 publications

(508 citation statements)

References 158 publications

(277 reference statements)

Supporting

Mentioning

438

Contrasting

Unclassified

Order By: Relevance

“…29 Despite the fact that there is no universal criteria for the simple ranking of different methods in terms of the degree of resemblance to realistic images, FID is widely used as a GAN performance evaluation metric. 31,35,36 For our purposes, FID was suitable for showing an improvement in the quality levels of images reconstructed with GAN-based methods compared to non-GAN-based methods. The FID scores in our study were slightly higher than those on other tasks, such as CIFAR-10, CelebA, and MNIST, among others 31,35 ; this is presumably related to differences in the sample sizes.…”

Section: Discussionmentioning

confidence: 98%

“…Despite the fact that there is no universal criteria for the simple ranking of different methods in terms of the degree of resemblance to realistic images, FID is widely used as a GAN performance evaluation metric . For our purposes, FID was suitable for showing an improvement in the quality levels of images reconstructed with GAN‐based methods compared to non‐GAN‐based methods.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Reconstruction of multicontrast MR images through deep learning

et al. 2020

View full text Add to dashboard Cite

Purpose Magnetic resonance (MR) imaging with a long scan time can lead to degraded images due to patient motion, patient discomfort, and increased costs. For these reasons, the role of rapid MR imaging is important. In this study, we propose the joint reconstruction of multicontrast brain MR images from down‐sampled data to accelerate the data acquisition process using a novel deep‐learning network. Methods Twenty‐one healthy volunteers (female/male = 7/14, age = 26 ± 4 yr, range 22–35 yr) and 16 postoperative patients (female/male = 7/9, age = 49 ± 9 yr, range 37–62 yr) were scanned on a 3T whole‐body scanner for prospective and retrospective studies, respectively, using both T1‐weighted spin‐echo (SE) and T2‐weighted fast spin‐echo (FSE) sequences. We proposed a network which we term “X‐net” to reconstruct both T1‐ and T2‐weighted images from down‐sampled images as well as a network termed “Y‐net” which reconstructs T2‐weighted images from highly down‐sampled T2‐weighted images and fully sampled T1‐weighted images. Both X‐net and Y‐net are composed of two concatenated subnetworks. We investigate optimal sampling patterns, the optimal patch size for augmentation, and the optimal acceleration factors for network training. An additional Y‐net combined with a generative adversarial network (GAN) was also implemented and tested to investigate the effects of the GAN on the Y‐net performance. Single‐ and joint‐reconstruction parallel‐imaging and compressed‐sensing algorithms along with a conventional U‐net were also tested and compared with the proposed networks. For this comparison, the structural similarity (SSIM), normalized mean square error (NMSE), and Fréchet inception distance (FID) were calculated between the outputs of the networks and fully sampled images. The statistical significance of the performance was evaluated by assessing the interclass correlation and in paired t‐tests. Results The outputs from the two concatenated subnetworks were closer to the fully sampled images compared to those from one subnetwork, with this result showing statistical significance. Uniform down‐sampling led to a statically significant improvement in the image quality compared to random or central down‐sampling patterns. In addition, the proposed networks provided higher SSIM and NMSE values than U‐net, compressed‐sensing, and parallel‐imaging algorithms, all at statistically significant levels. The GAN‐based Y‐net showed a better FID and more realistic images compared to a non‐GAN‐based Y‐net. The performance capabilities of the networks were similar between normal subjects and patients. Conclusions The proposed X‐net and Y‐net effectively reconstructed full images from down‐sampled images, outperforming the conventional parallel‐imaging, compressed‐sensing and U‐net methods and providing more realistic images in combination with a GAN. The developed networks potentially enable us to accelerate multicontrast anatomical MR imaging in routine clinical studies including T1‐and T2‐weighted imaging.

show abstract

Section: Discussionmentioning

confidence: 98%

Section: Discussionmentioning

confidence: 99%

Reconstruction of multicontrast MR images through deep learning

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Despite progress in the theoretical understanding of generative models and increased attention in GAN research, evaluating and comparing the performance of these models still remain a hard task. While several measures have been introduced, there is no consensus yet as to which measure best captures the strengths and limitations of the models and yield a fair model comparison [53]. Moreover, evaluation metrics are usually problem specific.…”

Section: Kinematic Evaluation Of Synthetic Signaturesmentioning

confidence: 99%

Motion Classification Using Kinematically Sifted ACGAN-Synthesized Radar Micro-Doppler Signatures

Erol

Gürbüz

Amin

2020

IEEE Trans. Aerosp. Electron. Syst.

View full text Add to dashboard Cite

Deep neural networks (DNNs) have recently received vast attention in applications requiring classification of radar returns, including radar-based human activity recognition for security, smart homes, assisted living, and biomedicine. However, acquiring a sufficiently large training dataset remains a daunting task due to the high human costs and resources required for radar data collection. In this paper, an extended approach to adversarial learning is proposed for generation of synthetic radar micro-Doppler signatures that are well-adapted to different environments. The synthetic data is evaluated using visual interpretation, analysis of kinematic consistency, data diversity, dimensions of the latent space, and saliency maps. A principlecomponent analysis (PCA) based kinematic-sifting algorithm is introduced to ensure that synthetic signatures are consistent with physically possible human motions. The synthetic dataset is used to train a 19-layer deep convolutional neural network (DCNN) to classify micro-Doppler signatures acquired from an environment different from that of the dataset supplied to the adversarial network. An overall accuracy 93% is achieved on a dataset that contains multiple aspect angles (0 • , 30 • , and 45 • as well as 60 • ), with 9% improvement as a result of kinematic sifting.B. Erol is currently with Siemens Corporate Technology, Princeton, NJ. Formerly, he was with the Center for Advanced Communications, Villanova University, PA, email: baris.erol@siemens.com.S.Z. Gurbuz is with the

show abstract

“…We train the CGAN for a total of 45 epochs, gradually increasing the training batch size from 32 to 256. In general, quantitative evaluation of GAN-generated images is a topic of ongoing debate with no clear consensus (Borji, 2018). We performed the training using a single Nvidia Tesla K80 general-purpose graphics processing unit; the full training required approximately 40 hr.…”

Section: 1029/2019gl082532mentioning

confidence: 99%

Reconstruction of Cloud Vertical Structure With a Generative Adversarial Network

Leinonen

Guillaume

Yuan

2019

Geophysical Research Letters

View full text Add to dashboard Cite

We demonstrate the feasibility of solving atmospheric remote sensing problems with machine learning using conditional generative adversarial networks (CGANs), implemented using convolutional neural networks. We apply the CGAN to generating two‐dimensional cloud vertical structures that would be observed by the CloudSat satellite‐based radar, using only the collocated Moderate‐Resolution Imaging Spectrometer measurements as input. The CGAN is usually able to generate reasonable guesses of the cloud structure and can infer complex structures such as multilayer clouds from the Moderate‐Resolution Imaging Spectrometer data. This network, which is formulated probabilistically, also estimates the uncertainty of its own predictions. We examine the statistics of the generated data and analyze the response of the network to each input parameter. The success of the CGAN in solving this problem suggests that generative adversarial networks are applicable to a wide range of problems in atmospheric science, a field characterized by complex spatial structures and observational uncertainties.

show abstract

Pros and cons of GAN evaluation measures

Cited by 720 publications

References 158 publications

Reconstruction of multicontrast MR images through deep learning

Reconstruction of multicontrast MR images through deep learning

Motion Classification Using Kinematically Sifted ACGAN-Synthesized Radar Micro-Doppler Signatures

Reconstruction of Cloud Vertical Structure With a Generative Adversarial Network

Contact Info

Product

Resources

About