Automated Generation of Accurate & Fluent Medical X-ray Reports

Abstract

Our paper aims to automate the generation of medical reports from chest X-ray image inputs, a critical yet time-consuming task for radiologists. Existing medical report generation efforts emphasize producing human-readable reports, yet the generated text may not be well aligned to the clinical facts. Our generated medical reports, on the other hand, are fluent and, more importantly, clinically accurate. This is achieved by our fully differentiable and end-to-end paradigm that contains three complementary modules: taking the chest X-ray images and clinical history document of patients as inputs, our classification module produces an internal checklist of disease-related topics, referred to as enriched disease embedding; the embedding representation is then passed to our transformer-based generator, to produce the medical report; meanwhile, our generator also creates a weighted embedding representation, which is fed to our interpreter to ensure consistency with respect to diseaserelated topics. Empirical evaluations demonstrate very promising results achieved by our approach on commonly-used metrics concerning language fluency and clinical accuracy. Moreover, noticeable performance gains are consistently observed when additional input information is available, such as the clinical document and extra scans from different views.* indicates equal contribution.

Recent advances of Transformers in medical image analysis: A comprehensive review

“…The Transformer‐based attempts to attempted to explore the automated generation of accurate and fluent X‐ray reports included 217–222 . As for the X‐ray medical report generation, You et al 219 build an AlignTransformer to reach the correspondence between the visual regions with the disease tags.…”

“…The Transformer-based attempts to attempted to explore the automated generation of accurate and fluent X-ray reports included. [217][218][219][220][221][222] As for the X-ray medical report generation, You et al 219 build an AlignTransformer to reach the correspondence between the visual regions with the disease tags. The authors divide the task into two parts: the first part is the prediction of disease tags and the feature extraction of the relation between the images and the disease tags while the second part is to produce the medical report based on the extracted information.…”

Vision Transformer and Language Model Based Radiology Report Generation

“…This model was unsupervised because it may be trained using diverse sets of photos and reports. Nguyen et al [18] proposed Classification of Clinical history and Chest X-ray to generate embedding of diseases along with a Transformer decoder sub-modules in an a fully differentiable paradigm to generate complete diagnostic reports. To ensure consistency with disease related topics, a weighted embedding representation was fed to the interpreter.…”

A Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?

“…A very close framework was described by Liu et al [184], in which the authors proposed an unsupervised model knowl-edge graph auto-encoder which accepts independent sets of images and reports during training, and consists of three modules: the pre-constructed knowledge graph, that works as the shared latent space and aims to bridge the visual and textual domains; the knowledge-driven encoder which projects medical images and reports to the corresponding coordinates in that latent space; and the knowledge-driven decoder that generates a medical report given a coordinate in that latent space. This modular structure is also employed by Nguyen et al [185], which added three complementary modules: a CNN-based classification module that produces an internal checklist of disease-related topics (i.e., the enriched disease embedding); a Transformer-based generator that generates the medical reports from the enriched disease embedding and produces a weighted embedding representation; and an interpreter that uses the weighted embedding representation to ensure consistency concerning disease-related topics. Similarly, You et al [186] proposed a framework, which includes two different attention-based modules: the align hierarchical attention module that first predicts the disease tags from the input image and then learns the multi-grained visual features by hierarchically aligning the visual regions and disease tags; and the multi-grained Transformer module that uses the multi-grained features to generate the medical reports.…”