A dataset of clinically generated visual questions and answers about radiology images

Abstract

Radiology images are an essential part of clinical decision making and population screening, e.g., for cancer. Automated systems could help clinicians cope with large amounts of images by answering questions about the image contents. An emerging area of artificial intelligence, Visual Question Answering (VQA) in the medical domain explores approaches to this form of clinical decision support. Success of such machine learning tools hinges on availability and design of collections composed of medical images augmented with question-answer pairs directed at the content of the image. We introduce VQA-RAD, the first manually constructed dataset where clinicians asked naturally occurring questions about radiology images and provided reference answers. Manual categorization of images and questions provides insight into clinically relevant tasks and the natural language to phrase them. Evaluating with well-known algorithms, we demonstrate the rich quality of this dataset over other automatically constructed ones. We propose VQA-RAD to encourage the community to design VQA tools with the goals of improving patient care.

Overcoming Data Limitation in Medical Visual Question Answering

“…VQA accuracy (%) Open-ended Close-ended VGG-16 (finetuning) [10] 24 CDAE, by firstly pretraining as described in Section 4.2, then finetuning, the finetuning significantly improves the performance over the training from scratch using only VQA-RAD. In addition, the results also show that our pretraining and finetuning of MAML and CDAE give better performance than the finetuning of VGG-16 which is pretrained on the ImageNet dataset.…”

“…Image feature f v and question embedding f q are fed into an attention mechanism (BAN [9] or SAN [19]) to produce a joint representation f a . This feature f a is used as input for a multi-class classifier (over the set of predefined answer classes [10]). To train the proposed model, we introduce a multi-task loss function to incorporate the effectiveness of the CDAE to VQA.…”

“…The VQA-RAD [10] dataset contains 315 images and 3,515 corresponding questions. Each image is associated with more than one question.…”

“…We compare our framework (Figure 1) with the baselines in [10]. In [10], the authors report the results when applying the general VQA frameworks, i.e., SAN framework [19], MCB framework [5] 6 and finetuning on VQA-RAD dataset. We also report another strong baseline when finetuning the state-of-the-art BAN framework [9].…”

“…By that construction, the data has high noisy level, i.e., the dataset includes many images that are not useful for direct patient care and it also contains questions that do not make any sense. Recently, in [10], the authors released the first manually constructed dataset VQA-RAD for medical VQA. Unfortunately, it contains only 315 images, which prevents to directly apply the powerful deep learning models for the VQA problem.…”