Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports

Jing, Baoyu; Wang, Zeya; Xing, Eric P.

doi:10.18653/v1/p19-1657

Cited by 108 publications

(129 citation statements)

References 30 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…Impression No acute cardiopulmonary abnormality. in this area Johnson et al, 2019;Liu et al, 2019;Jing et al, 2019).…”

Section: Resultsmentioning

confidence: 99%

“…In addition, we also compare our model with those in previous studies, including conventional image captioning models, e.g., ST (Vinyals et al, 2015), ATT2IN (Rennie et al, 2017), ADAATT (Lu et al, 2017), TOPDOWN (Anderson et al, 2018), and the ones proposed for the medical domain, e.g., COATT , HRGR and CMAS-RL (Jing et al, 2019).…”

Section: Baseline and Evaluation Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Generating Radiology Reports via Memory-driven Transformer

Chen¹,

Yan²,

Chang³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

257

192

View full text Add to dashboard Cite

Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radiologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an essential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memorydriven Transformer, where a relational memory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our approach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings. 1

show abstract

“…Impression No acute cardiopulmonary abnormality. in this area Johnson et al, 2019;Liu et al, 2019;Jing et al, 2019).…”

Section: Resultsmentioning

confidence: 99%

Section: Baseline and Evaluation Metricsmentioning

confidence: 99%

Generating Radiology Reports via Memory-driven Transformer

Chen¹,

Yan²,

Chang³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

257

192

View full text Add to dashboard Cite

show abstract

“…This tendency results from an imbalanced frequency of findings among the medical images. Jing et al (2019) and Harzig et al (2019) use different decoders to generate normalities or abnormalities to address these data imbalance difficulties. Biswal et al (2020) accepts doctors' anchor words for controllable medical report generation.…”

Section: Related Workmentioning

confidence: 99%

“…Our work addresses the data imbalance difficulties beyond the imbalance between normalities and abnormalities, as Jing et al (2019) addressed. Moreover, with our approach, the doctors can reflect their intentions to reports more directly to a greater degree than Biswal et al (2020).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning with Imbalanced Dataset for Data-to-Text Medical Report Generation

Nishino¹,

Ozaki²,

Momoki³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Automated generation of medical reports that describe the findings in the medical images helps radiologists by alleviating their workload. Medical report generation system should generate correct and concise reports. However, data imbalance makes it difficult to train models accurately. Medical datasets are commonly imbalanced in their finding labels because incidence rates differ among diseases; moreover, the ratios of abnormalities to normalities are significantly imbalanced. We propose a novel reinforcement learning method with a reconstructor to improve the clinical correctness of generated reports to train the data-to-text module with a highly imbalanced dataset. Moreover, we introduce a novel data augmentation strategy for reinforcement learning to additionally train the model on infrequent findings. From the perspective of a practical use, we employ a Two-Stage Medical Report Generator (TS-MRGen) for controllable report generation from input images. TS-MRGen consists of two separated stages: an image diagnosis module and a data-to-text module. Radiologists can modify the image diagnosis module results to control the reports that the data-totext module generates. We conduct an experiment with two medical datasets to assess the data-to-text module and the entire two-stage model. Results demonstrate that the reports generated by our model describe the findings in the input image more correctly.

show abstract