Towards Visual Dialog for Radiology

Abstract

Current research in machine learning for radiology is focused mostly on images. There exists limited work in investigating intelligent interactive systems for radiology. To address this limitation, we introduce a realistic and information-rich task of Visual Dialog in radiology, specific to chest X-ray images. Using MIMIC-CXR, an openly available database of chest X-ray images, we construct both a synthetic and a real-world dataset and provide baseline scores achieved by state-of-theart models. We show that incorporating medical history of the patient leads to better performance in answering questions as opposed to conventional visual question answering model which looks only at the image. While our experiments show promising results, they indicate that the task is extremely challenging with significant scope for improvement. We make both the datasets (synthetic and gold standard) and the associated code publicly available to the research community.

Medical Visual Question Answering: A Survey

“…To the best of our knowledge, there are 7 public-available medical VQA datasets up to date: VQA-MED-2018 [19], VQA-RAD [32], VQA-MED-2019 [10], RadVisDial [31], PathVQA [21], VQA-MED-2020 [9] and SLAKE [37](in chronological order). The datasets details are summarized in Table 1.…”

“…RadVisDial [31] is the first publicly available dataset for visual dialog in radiology. The visual dialogue consists of multiple QA pairs and is considered a more practical and complicated task to a radiology AI system than VQA.…”

“…The medical VQA tasks are restricted to certain subjects, such as radiology and pathology. Some datasets [10,31,32] further restrict the images to certain modalities and organ systems. The questions in medical VQA are usually about one object or its attribute.…”

“…In other datasets such as VQA-Med-2019 [10] and PathVQA [21], the question categories tend to be less diverse than the VQA-RAD dataset. In the RadVisDial [31] and VQA-Med-2020 [9], the question category is restricted to abnormality presence. However, most questions about an abnormality in existing datasets are about presence without further inquiry like the location of tumors or tumor size.…”

“…Inspired by the VQA research in the general domain, the recent exploration of medical VQA has attracted great interest. The medical VQA system is expected to assist in clinical decision making and improve patient engagement [19,31]. Unlike other medical AI applications often restricted to pre-defined diseases or organ types, the medical VQA can understand free-form questions in natural language and provide reliable and user-friendly answers.…”