Clinically Accurate Chest X-Ray Report Generation

Liu, Guanxiong; Hsu, Tzu-Ming Harry; McDermott, Matthew; Boag, Willie; Weng, Wei-Hung; Szolovits, Peter; Ghassemi, Marzyeh

doi:10.48550/arxiv.1904.02633

Cited by 17 publications

(32 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dataset contains frontal (anteroposterio (AP), posteroanterior (PA)), and lateral view images. Some previous work makes no distinction between view positions [7,23], but this might lead to a miss-match between some findings between an image and report pair. For example, cardiomegaly can only be accurately assessed in the PA view, while the AP view will exaggerate the heart silhouette due to magnification [28].…”

Section: Datasetsmentioning

confidence: 99%

“…Tienet [36] is a pioneering CNN-RNN model with image-text attention mechanism for VLU (e.g., thorax disease classification) and VLG (e.g., report generation) tasks by using ChestXray14 [35] dataset. Liu et al [23] only focuses on the VLG task to generate the radiology report utilizing a CNN-RNN-RNN architecture with a hierarchical generation strategy from MIMIC-CXR [17] and Open-I dataset [11]. Hsu et al [14] focuses on a VLU task, specifically medical image-report retrieval in the MIMIC-CXR dataset, based on supervised and unsupervised methods.…”

Section: Related Workmentioning

confidence: 99%

“…All models, including the re-implemented TieNet and our models, use beam search with a beam size of 4. The source code of TieNet [36] is not publicly opened, so we re-implemented Tienet by referring to the original paper and Liu et al [23] using Pytorch. TieNet consist of 4 modules: encoder, decoder, AETE, and joint layer.…”

Section: A Appendixmentioning

confidence: 99%

See 2 more Smart Citations

Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

Moon¹,

Lee²,

Shin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL) which adopts a Transformer-based architecture combined with a novel multimodal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (image-report retrieval, disease classification, medical visual question answering) and vision-language generation task (report generation). By rigorously evaluating the proposed model on four downstream tasks with two chest X-ray image datasets (MIMIC-CXR and Open-I), we empirically demonstrate the superior downstream task performance of MedViLL against various baselines including task-specific architectures.

show abstract

Section: Datasetsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

Moon¹,

Lee²,

Shin³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, studies on medical image captioning also have been active [8][9][10]. Baoyu et al [11] introduced a co-attention (CoATT) mechanism for localizing sub-regions in the In addition, the CEDT uses all levels of information of the encoders in order not to lose features extracted from the GLVE when decoding phrases.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, studies on medical image captioning also have been active [8][9][10]. Baoyu et al [11] introduced a co-attention (CoATT) mechanism for localizing sub-regions in the image and generating the corresponding descriptions, as well as a hierarchical LSTM model to generate long paragraphs.…”

Section: Related Workmentioning

confidence: 99%

Cross Encoder-Decoder Transformer with Global-Local Visual Extractor for Medical Image Captioning

Lee

Cho

Park

et al. 2022

Sensors

View full text Add to dashboard Cite

Transformer-based approaches have shown good results in image captioning tasks. However, current approaches have a limitation in generating text from global features of an entire image. Therefore, we propose novel methods for generating better image captioning as follows: (1) The Global-Local Visual Extractor (GLVE) to capture both global features and local features. (2) The Cross Encoder-Decoder Transformer (CEDT) for injecting multiple-level encoder features into the decoding process. GLVE extracts not only global visual features that can be obtained from an entire image, such as size of organ or bone structure, but also local visual features that can be generated from a local region, such as lesion area. Given an image, CEDT can create a detailed description of the overall features by injecting both low-level and high-level encoder outputs into the decoder. Each method contributes to performance improvement and generates a description such as organ size and bone structure. The proposed model was evaluated on the IU X-ray dataset and achieved better performance than the transformer-based baseline results, by 5.6% in BLEU score, by 0.56% in METEOR, and by 1.98% in ROUGE-L.

show abstract

Better Understanding of the Metamorphosis of Pregnancy (BUMP): protocol for a digital feasibility study in women from preconception to postpartum

et al. 2022

View full text Add to dashboard Cite

The Better Understanding the Metamorphosis of Pregnancy (BUMP) study is a longitudinal feasibility study aimed to gain a deeper understanding of the pre-pregnancy and pregnancy symptom experience using digital tools. The present paper describes the protocol for the BUMP study. Over 1000 participants are being recruited through a patient provider-platform and through other channels in the United States (US). Participants in a preconception cohort (BUMP-C) are followed for 6 months, or until conception, while participants in a pregnancy cohort (BUMP) are followed into their fourth trimester. Participants are provided with a smart ring, a smartwatch (BUMP only), and a smart scale (BUMP only) alongside cohort-specific study apps. Participant centric engagement strategies are used that aim to co-design the digital approach with participants while providing knowledge and support. The BUMP study is intended to lay the foundational work for a larger study to determine whether participant co-designed digital tools can be used to detect, track and return multimodal symptoms during the perinatal window to inform individual level symptom trajectories.

show abstract

Clinically Accurate Chest X-Ray Report Generation

Cited by 17 publications

References 30 publications

Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

Cross Encoder-Decoder Transformer with Global-Local Visual Extractor for Medical Image Captioning

Better Understanding of the Metamorphosis of Pregnancy (BUMP): protocol for a digital feasibility study in women from preconception to postpartum

Contact Info

Product

Resources

About