Implicit Embeddings via GAN Inversion for High Resolution Chest Radiographs

Weber, Tobias; Ingrisch, Michael; Bischl, Bernd; Rügamer, David

doi:10.1007/978-3-031-25046-0_3

Cited by 3 publications

(3 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We observe that employing image embedding as input leads to higher scores in both natural language metrics and clinical efficacy metrics compared to using raw images, suggesting that the encoding process meaningfully compresses image information, emphasizing relevant details crucial for feature extraction and recognition by the prior model. Notably, this is consistent with the observation in image generation tasks reported by Weber et al [43], who concluded that semantic features are more beneficial for a cascaded diffusion model in generating high-quality and high-resolution CXR images compared to low-resolution images.…”

Section: Image Vs Image Embeddingsupporting

confidence: 92%

“…For this purpose, deep generative models are utilized to augment the CXR image dataset. Previous studies have demonstrated the generation of CXR images using deep generative models, including generative adversarial networks (GANs) and diffusion models [2,3,5,6,19,21,27,31,43]. CXR images are typically annotated with radiology reports detailing clinical observations made by radiologists, as depicted in Fig.…”

Section: Introductionmentioning

confidence: 99%

“…Packhäuser et al [33] verified the performance of LDM in generating high-quality CXR images, and found that the images generated by LDM outperform those by PGGAN in an abnormality identification task. Weber et al [43] proposed a cascaded LDM Cheff that can generate high-quality CXR images on a 1-megapixel scale. Based on the conclusions drawn by Chambon et al [5,6], we adopt a pre-trained LDM as the backbone of the vision module, and attempt methods to further improve generation quality.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

CXR-IRGen: An Integrated Vision and Language Model for the Generation of Clinically Accurate Chest X-Ray Image-Report Pairs

Shentu,

Al Moubayed

2024

2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Chest X-Ray (CXR) images play a crucial role in clinical practice, providing vital support for diagnosis and treatment. Augmenting the CXR dataset with synthetically generated CXR images annotated with radiology reports can enhance the performance of deep learning models for various tasks. However, existing studies have primarily focused on generating unimodal data of either images or reports. In this study, we propose an integrated model, CXR-IRGen, designed specifically for generating CXR image-report pairs. Our model follows a modularized structure consisting of a vision module and a language module. Notably, we present a novel prompt design for the vision module by combining both text embedding and image embedding of a reference image. Additionally, we propose a new CXR report generation model as the language module, which effectively leverages a large language model and self-supervised learning strategy. Experimental results demonstrate that our new prompt is capable of improving the general quality (FID) and clinical efficacy (AUROC) of the generated images, with average improvements of 15.84% and 1.84%, respectively. Moreover, the proposed CXR report generation model outperforms baseline models in terms of clinical efficacy (F 1 score) and exhibits a high-level alignment of image and text, as the best F 1 score of our model is 6.93% higher than the state-of-the-art CXR report generation model. Our code is available at https://github.com/junjie-shentu/CXR-IRGen.

show abstract

Section: Image Vs Image Embeddingsupporting

confidence: 92%