MedSkip: Medical Report Generation Using Skip Connections and Integrated Attention

Pahwa, Esha; Mehta, Dwij; Kapadia, Sanjeet; Jain, Devansh; Luthra, Achleshwar

doi:10.1109/iccvw54120.2021.00380

Cited by 12 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We first compare our model with the SOTA medical report generation models CO-ATT [11], CMAS-RL [10], HRGR [18], R2Gen [5], R2GenCMN [4], PPKED [21], KERP [17], XproNet [33], Med-Skip [24], and CA [22]. Since the models PPKED, KERP, Med-Skip, and CA are not open-sourced, we directly quote the results published in their literature.…”

Section: Resultsmentioning

confidence: 99%

“…HRGR * [18] 0.438 0.298 0.208 0.151 0.322 --CO-ATT * [11] 0.455 0.288 0.205 0.154 0.369 --CMAS-RL * [10] 0.464 0.301 0.210 0.154 0.362 --R2Gen * * * [5] 0.458 0.295 0.210 0.159 0.375 0.176 0.408 MedSkip * * [24] 0.467 0.297 0.214 0.162 0.355 0.187 -KERP * * [17] 0.470 0.304 0.219 0.165 0.371 0.187 0.280 PPKED * * [21] 0 the literature, and could further benefit the latter. On comparing the results of our approach ITHN with that of MoCHi, we could observe an average relative increase of +6.6% in BLUE, +7.1% in METEOR, and +20% in CIDER for the best performing model XproNet on IU-XRay dataset.…”

Section: Quantitative Analysismentioning

confidence: 99%

See 1 more Smart Citation

Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives

Voutharoja,

Wang,

Zhou

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Automatic radiology report generation is challenging as medical images or reports are usually similar to each other due to the common content of anatomy. This makes a model hard to capture the uniqueness of individual images and is prone to producing undesired generic or mismatched reports. This situation calls for learning more discriminative features that could capture even fine-grained mismatches between images and reports. To achieve this, this paper proposes a novel framework to learn discriminative image and report features by distinguishing them from their closest peers, i.e., hard negatives. Especially, to attain more discriminative features, we gradually raise the difficulty of such a learning task by creating increasingly hard negative reports for each image in the feature space during training, respectively. By treating the increasingly hard negatives as auxiliary variables, we formulate this process as a min-max alternating optimisation problem. At each iteration, conditioned on a given set of hard negative reports, image and report features are learned as usual by minimising the loss functions related to report generation. After that, a new set of harder negative reports will be created by maximising a loss reflecting image-report alignment. By solving this optimisation, we attain a model that can generate more specific and accurate reports. It is noteworthy that our framework enhances discriminative feature learning without introducing extra network weights. Also, in contrast to the existing way of generating hard negatives, our framework extends beyond the granularity of the dataset by generating harder samples out of the training set. Experimental study on benchmark datasets verifies the efficacy of our framework and shows that it can serve as a plug-in to readily improve existing medical report generation models. The code is publicly available at https://github.com/Bhanu068/ITHN.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Quantitative Analysismentioning

confidence: 99%

Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives

Voutharoja,

Wang,

Zhou

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

show abstract

“…Image captioning is a traditional task and has received extensive research interest (You et al, 2016;Aneja et al, 2018;Xu et al, 2021). Radiology report generation can be treated as an extension of image captioning tasks to the medical domain, aiming to describe radiology images in the text (i.e., findings), and has achieved considerable improvements in recent years (Chen et al, 2020;Zhang et al, 2020a;Liu et al, 2019bLiu et al, , 2021bZhou et al, 2021;Boag et al, 2020;Pahwa et al, 2021;Jing et al, 2019;Zhang et al, 2020b;You et al, 2021;Liu et al, 2019a). Liu et al (2021a) employed competence-based curriculum learning to promote report generation, which started from simple reports and then attempted to consume harder reports.…”

Section: Radiology Report Generationmentioning

confidence: 99%

Improving Radiology Summarization with Radiograph and Anatomy Prompts

Hu¹,

Chen²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The impression is crucial for the referring physicians to grasp key information since it is concluded from the findings and reasoning of radiologists. To alleviate the workload of radiologists and reduce repetitive human labor in impression writing, many researchers have focused on automatic impression generation. However, recent works on this task mainly summarize the corresponding findings and pay less attention to the radiology images. In clinical, radiographs can provide more detailed valuable observations to enhance radiologists' impression writing, especially for complicated cases. Besides, each sentence in findings usually focuses on single anatomy, so they only need to be matched to corresponding anatomical regions instead of the whole image, which is beneficial for textual and visual features alignment. Therefore, we propose a novel anatomy-enhanced multimodal model to promote impression generation. In detail, we first construct a set of rules to extract anatomies and put these prompts into each sentence to highlight anatomy characteristics. Then, two separate encoders are applied to extract features from the radiograph and findings. Afterward, we utilize a contrastive learning module to align these two representations at the overall level and use a co-attention to fuse them at the sentence level with the help of anatomyenhanced sentence representation. Finally, the decoder takes the fused information as the input to generate impressions. The experimental results on two benchmark datasets confirm the effectiveness of the proposed method, which achieves state-of-the-art results.

show abstract

Recent advances of Transformers in medical image analysis: A comprehensive review

Xia

Wang

2023

MedComm – Future Medicine

View full text Add to dashboard Cite

Recent works have shown that Transformer's excellent performances on natural language processing tasks can be maintained on natural image analysis tasks. However, the complicated clinical settings in medical image analysis and varied disease properties bring new challenges for the use of Transformer. The computer vision and medical engineering communities have devoted significant effort to medical image analysis research based on Transformer with especial focus on scenario-specific architectural variations.In this paper, we comprehensively review this rapidly developing area by covering the latest advances of Transformer-based methods in medical image analysis of different settings. We first give introduction of basic mechanisms of Transformer including implementations of selfattention and typical architectures. The important research problems in various medical image data modalities, clinical visual tasks, organs and diseases are then reviewed systemically. We carefully collect 276 very recent works and 76 public medical image analysis datasets in an organized structure. Finally, discussions on open problems and future research directions are also provided. We expect this review to be an up-to-date roadmap and serve as a reference source in pursuit of boosting the development of medical image analysis field.

show abstract

MedSkip: Medical Report Generation Using Skip Connections and Integrated Attention

Cited by 12 publications

References 15 publications

Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives

Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives

Improving Radiology Summarization with Radiograph and Anatomy Prompts

Recent advances of Transformers in medical image analysis: A comprehensive review

Contact Info

Product

Resources

About