Preparing a collection of radiology examinations for distribution and retrieval

Demner‐Fushman, Dina; Kohli, Marc D.; Rosenman, Marc B.; Shooshan, Sonya E.; Rodriguez, Laritza; Antani, Sameer; Thoma, George R.; McDonald, Clement J.

doi:10.1093/jamia/ocv080

Cited by 641 publications

(381 citation statements)

References 13 publications

Supporting

Mentioning

379

Contrasting

Unclassified

Order By: Relevance

“…We apply a multi-task scheme including: (1) radiographic observation classification to pretrain and fine-tune the encoder with large-scale images; (2) to extract medical concepts; and (3) to fuse all information to generate radiology reports. Therefore, two datasets are used in this work: CheXpert [5], a large collection of chest x-ray images under 14 common chest radiographic observations to pretrain the image encoder, and Indiana University Chest X-ray [1] containing full radiology reports but in a considerably smaller scale for training and evaluating the report generation task.…”

Section: Methodsologymentioning

confidence: 99%

“…Since neither of the aforementioned datasets released radiology reports, we use IU-RR [1] for evaluating radiology report generation. For preprocessing, we first removed samples without multi-view images, and concatenated the "findings" and "impression" sections because in some forms all contents are either in the "findings" or "impression" section with the other left blank.…”

Section: Methodsmentioning

confidence: 99%

“…Additionally, we consider both frontal and lateral images of one patient as one input pair and enforce the prediction consistencies under different views by a mean square error (MSE) loss over the multi-view encoder outputs. The loss function is thus defined in Equation 1 where y i,j is the j-th ground truth entry (j ∈ [1,14])) of the i-th sample (i ∈ (1, N )), the frontal view and lateral view prediction are denoted asŷ i f and y i l . The encoder outputs global features after average-pooling, and local features v ∈ R k×dv from the last CNN block where k denotes the number of local regions and d v denotes the dimension.…”

Section: Image Encodermentioning

confidence: 99%

“…We choose the widely used Indiana University Chest X-ray radiology report dataset (IU-RR) [1] for this task. In most cases, radiology reports contain descriptive findings in the form of paragraphs, and conclusive impressions in one or a few sentences.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

Yuan

Liao

Luo

et al. 2019

Lecture Notes in Computer Science

145

106

View full text Add to dashboard Cite

Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.

show abstract

Section: Methodsologymentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Image Encodermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

Yuan

Liao

Luo

et al. 2019

Lecture Notes in Computer Science

145

106

View full text Add to dashboard Cite

show abstract

“…EURORAD, which is operated by the European Society of Radiology also includes but is not limited to cancer images. Nevertheless, it mainly focuses on the training of radiologists and provides no automated data access 3 Open-i is a service hosted by the National Library of Medicine (NLM) [130]. It provides a search engine and a download API for accessing images from PubMed Central articles, NLM History of Medicine collection and other sources.…”

Section: Data Sharingmentioning

confidence: 99%

Personalizing Medicine Through Hybrid Imaging and Medical Big Data Analysis

et al. 2018

View full text Add to dashboard Cite

Medical imaging has evolved from a pure visualization tool to representing a primary source of analytic approaches toward in vivo disease characterization. Hybrid imaging is an integral part of this approach, as it provides complementary visual and quantitative information in the form of morphological and functional insights into the living body. As such, non-invasive imaging modalities no longer provide images only, but data, as stated recently by pioneers in the field. Today, such information, together with other, non-imaging medical data creates highly heterogeneous data sets that underpin the concept of medical big data. While the exponential growth of medical big data challenges their processing, they inherently contain information that benefits a patient-centric personalized healthcare. Novel machine learning approaches combined with high-performance distributed cloud computing technologies help explore medical big data. Such exploration and subsequent generation of knowledge require a profound understanding of the technical challenges. These challenges increase in complexity when employing hybrid, aka dual-or even multi-modality image data as input to big data repositories. This paper provides a general insight into medical big data analysis in light of the use of hybrid imaging information. First, hybrid imaging is introduced (see further contributions to this special Research Topic), also in the context of medical big data, then the technological background of machine learning as well as state-of-the-art distributed cloud computing technologies are presented, followed by the discussion of data preservation and data sharing trends. Joint data exploration endeavors in the context of in vivo radiomics and hybrid imaging will be presented. Standardization challenges of imaging protocol, delineation, feature engineering, and machine learning evaluation will be detailed. Last, the paper will provide an outlook into the future role of hybrid imaging in view of personalized medicine, whereby a focus will be given to the derivation of prediction models as part of clinical decision support systems, to which machine learning approaches and hybrid imaging can be anchored.

show abstract