Medical Visual Question Answering: A Survey

Abstract

Medical Visual Question Answering (VQA) is a combination of medical artificial intelligence and popular VQA challenges. Given a medical image and a clinically relevant question in natural language, the medical VQA system is expected to predict a plausible and convincing answer. Although the general-domain VQA has been extensively studied, the medical VQA still needs specific investigation and exploration due to its task features. In the first part of this survey, we cover and discuss the publicly available medical VQA datasets up to date about the data source, data quantity, and task feature. In the second part, we review the approaches used in medical VQA tasks. In the last part, we analyze some medical-specific challenges for the field and discuss future research directions.

Fair Machine Learning in Healthcare: A Review

“…The combination of different modalities of healthcare data, each providing information about a patient's treatment from a specific perspective, overlays and complements each other to further improve the accuracy of diagnosis and treatment. For example, the visual quest answering (VQA) task [6] combines computer vision (CV) and natural language processing (NLP), and the model can answer relevant questions based on medical images and clinical notes [60]. However, multi-modal models face more serious bias and fairness issues than unimodal models, despite improvements in performance [12].…”

A Survey on Multi-hop Question Answering and Generation

“…A number of works have also attempted to decompose the multi-hop questions into single hop questions or generate follow-up questions based on the retrieved information [14,95,102,138,181]. single-hop QA [2,12,33,60,103,132], open-domain QA [184], medical QA [68,88] etc. The surveys that are most relevant to MHQA are the ones focused on QA over knowledge bases [32,43,82] and visual QA [88,135,164].…”

“…single-hop QA [2,12,33,60,103,132], open-domain QA [184], medical QA [68,88] etc. The surveys that are most relevant to MHQA are the ones focused on QA over knowledge bases [32,43,82] and visual QA [88,135,164]. However, these can be considered as sub-domains of the more general formulation of MHQA field that this manuscript aims to survey.…”

Is Unimodal Bias Always Bad for Visual Question Answering? A Medical Domain Study with Dynamic Attention

“…• We propose a novel pipeline to reinforce the text bias after fusing multimodal features, combined with dynamic attention, named the reinforCe unimOdal dynamiC Attention model (COCA), which can be applied universally for visual question answering tasks. • To the best of our knowledge, we are the first to question an overall accepted 'myth' that unimodal biases in medical VQA should be avoided [7] and, moreover, prove that adding unimodal bias after feature fusion under certain conditions can improve the prediction accuracy under specific circumstances. • Experimental results on a real-world dataset show the superior performance of the proposed COCA model compared with the state-of-the-art.…”

“…However, there are cases when reducing unimodal bias will not benefit model performance for other textually biased VQA datasets [15]. Due to the small scale of VQA datasets in the medical domain [7], this is less covered and some questions indeed do not need to view the related image before answering. In such cases, reinforcing unimodal biases can improve prediction performance, particularly after fusing multimodal features, which we study in this paper.…”