Contrastive Attention for Automatic Chest X-ray Report Generation

Abstract

Recently, chest X-ray report generation, which aims to automatically generate descriptions of given chest X-ray images, has received growing research interests. The key challenge of chest X-ray report generation is to accurately capture and describe the abnormal regions. In most cases, the normal regions dominate the entire chest X-ray image, and the corresponding descriptions of these normal regions dominate the final report. Due to such data bias, learning-based models may fail to attend to abnormal regions. In this work, to effectively capture and describe abnormal regions, we propose the Contrastive Attention (CA) model. Instead of solely focusing on the current input image, the CA model compares the current input image with normal images to distill the contrastive information. The acquired contrastive information can better represent the visual features of abnormal regions. According to the experiments on the public IU-X-ray and MIMIC-CXR datasets, incorporating our CA into several existing models can boost their performance across most metrics. In addition, according to the analysis, the CA model can help existing models better attend to the abnormal regions and provide more accurate descriptions which are crucial for an interpretable diagnosis. Specifically, we achieve the state-ofthe-art results on the two public datasets.

Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation

“…In recent years, automatic radiology report generation has attracted extensive research interests [45,22,40,14,6,29]. Most existing methods, like [15,42,44] follow the standard image captioning approaches and employ the encoder-decoder framework, e.g., CNN-HRNN [15,23].…”

“…Besides, many similar sentences are used to describe the same normal regions. With this unbalanced textual distribution, training with such dataset makes the generation of normal sentences dominant [14,42,44,29], disabling the model to describe specific crucial abnormalities. In brief, as shown in Figure 1, the widely-used HRNN [19] generates some repeated sentences of normalities and fails to depict some rare but important abnormalities.…”

“…To ensure these rare but important abnormal regions captured and described, the urgent problem is to alleviate such serious data deviation problem [35,22,14,20,45,29]. In our work, we propose the Posterior-and-Prior Knowledge Exploring-and-Distilling (PPKED) framework, which imitates the radiologists' working patterns to alleviate above problems.…”

“…Similar to image paragraph generation, most existing works [15,42,44] attempt to adopt a HRNN to automatically generate a fluent report. However, due to the serious data deviation, these models are poor at finding visual groundings and are biased towards generating plausible but general reports without prominent abnormal narratives [14,22,44,29].…”

“…Currently, some approaches [14,22,44,45,21,29] have been proposed to alleviate data deviation. In detail, instead of only adopting a single sentence-level RNN to generate both the normal and abnormal sentences, [14] introduced two RNNs as two different report writers, i.e., Normality Writer and Abnormality Writer, to help the model to generate more accurate normal and abnormal sentences, respectively.…”