BPI-MVQA: a bi-branch model for medical visual question answering

Abstract

Background Visual question answering in medical domain (VQA-Med) exhibits great potential for enhancing confidence in diagnosing diseases and helping patients better understand their medical conditions. One of the challenges in VQA-Med is how to better understand and combine the semantic features of medical images (e.g., X-rays, Magnetic Resonance Imaging(MRI)) and answer the corresponding questions accurately in unlabeled medical datasets. Method We propose a novel Bi-branched model based on Parallel networks and Image retrieval for Medical Visual Question Answering (BPI-MVQA). The first branch of BPI-MVQA is a transformer structure based on a parallel network to achieve complementary advantages in image sequence feature and spatial feature extraction, and multi-modal features are implicitly fused by using the multi-head self-attention mechanism. The second branch is retrieving the similarity of image features generated by the VGG16 network to obtain similar text descriptions as labels. Result The BPI-MVQA model achieves state-of-the-art results on three VQA-Med datasets, and the main metric scores exceed the best results so far by 0.2$$\%$$ % , 1.4$$\%$$ % , and 1.1$$\%$$ % . Conclusion The evaluation results support the effectiveness of the BPI-MVQA model in VQA-Med. The design of the bi-branch structure helps the model answer different types of visual questions. The parallel network allows for multi-angle image feature extraction, a unique feature extraction method that helps the model better understand the semantic information of the image and achieve greater accuracy in the multi-classification of VQA-Med. In addition, image retrieval helps the model answer irregular, open-ended type questions from the perspective of understanding the information provided by images. The comparison of our method with state-of-the-art methods on three datasets also shows that our method can bring substantial improvement to the VQA-Med system.

Counterfactual Causal-Effect Intervention for Interpretable Medical Visual Question Answering

“…Then, these models were transferred into a lightweight student model for fine-tuning radiological images of VQA-Med datasets. Another study [28] proposed a bi-branched model based on parallel networks and image retrieval for VQA-Med (BPI-MVQA) to realize complementary advantages in image sequence feature extraction, spatial feature extraction, and multi-modal fusion, forcing the VQA-Med to consider the feature applicability to specific image-understanding tasks [29].…”

“…We compare our CCIS-MVQA model with some SOTA methods: MEVF [25], CPRD [27], BPI-MVQA [28], CGMVQA [30], QC-MLB [32], QFPN [35], Caption-Aware [36], AOM [39], and Optimal Model [50]. We have reviewed these models in Section II.…”

Vision–Language Model for Visual Question Answering in Medical Imagery

“…The model predicts the answer either by a classification or a generation head depending on the type of question. Finally, in [ 50 ], a bibranched model is proposed in which the first branch answers closed-ended questions with a transformer architecture, and the second branch answers open-ended questions with image retrieval that gives the most similar answer to the test image.…”

A Comprehensive Review and Open Challenges on Visual Question Answering Models

“…A few of the difficulties that VQA-Med encounters include the requirement for special processing of medical-specific vocabulary in medical texts and images, a challenge in combining multi-modal features at various levels of medical texts and images, and a propensity to ignore the relationship between the question and the visual information deduced from the text semantics. The VQA model was presented by [61] with two branches. The model uses a transformer structure for the common classification problem, three embedding methods, a hierarchy of feature extractors, a parallel structure of GRU and ResNet152 as image feature extractors, and specialized segmentation symbols as input.…”