Surgical-VQA: Visual Question Answering in Surgical Scenes Using Transformer

Seenivasan, Lalithkumar; Islam, Mobarakol; Krishna, Adithya K.; Ren, Hongliang

doi:10.1007/978-3-031-16449-1_4

Cited by 15 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model of (8) employs vision-text transformers and a residual MLP-based VisualBERT encoder to improve performance in classification-based answering. A multi-modal transformer-based architecture, VB-Fusion, is proposed by (9) to learn joint representations by combining modality-specific features with multi-modal transformer layers.…”

Section: Transformer Based Approaches For Visual Question Answeringmentioning

confidence: 99%

“…By using the ViT architecture, this approach enables the model to perform vision-based tasks with high performance, achieving notable success in many vision tasks because of its ability to capture the long-range dependencies and complex visual patterns through self-attention. The (43) introduces the Language-Vision GPT (LV-GPT) model, which incorporates vision input for VQA in surgery. It utilizes ViT, ResNet18 and Swin for feature extraction.…”

Section: Vision Transformers For Visual Question Answeringmentioning

confidence: 99%

“…Seeing is Knowing (106) , MULAN (107) Faster R-CNN with ResNet-101 GAT (108) , ATH (109) , DMMGR (24) , MCLN (110) , MCAN (111) , F-SWAP (112) , SRRN (35) , TVQA (113) Faster R-CNN with Resnet-152 RA-MAP (114) , MASN (115) , Anamoly based (114) , Vocab based (116) , DA-Net (117) ResNet CNN within Faster R-CNN MuVAM (118) FasterR-CNN with ResNext-152 CBM (119) RCNN (120) Multi-image (89) VGGNet (121) VQA-AID (122) EfficientNetV2 (123) RealFormer (124) YOLO (125) Scene Text VQA (126) CLIPViT-B CCVQA (14) Resnet NFNet (127) Flamingo (128) ViT (129) VLMmed (46) , ConvS2S+ViT (130) , BMT (10) , M2I2 (52) XCLIP with ViT-L/14 CMQR (32) RsNet18, Swin, ViT LV-GPT (43) GLIP (131) REVIVE (132) CLIP (133) KVQAE (30) 2.6.4 VGGNet (121) VGGNet (Visual Geometry Group Network) is a CNN with a small number of layers, achieving good performance in image classification tasks. It is basically known for its simplicity and generalizability to new datasets.…”

Section: Faster Rcnnmentioning

confidence: 99%

See 2 more Smart Citations

A Review of Recent Advances in Visual Question Answering: Capsule Networks and Vision Transformers in Focus

Prakash,

Devananda

2024

IJST

View full text Add to dashboard Cite

Objectives: Multimodal deep learning, incorporating images, text, videos, speech, and acoustic signals, has grown significantly. This article aims to explore the untapped possibilities of multimodal deep learning in Visual Question Answering (VQA) and address a research gap in the development of effective techniques for comprehensive image feature extraction. Methods: This article provides a comprehensive overview of VQA and the associated challenges. It emphasizes the need for an extensive representation of images in VQA and pinpoints the specific research gap pertaining to image feature extraction and highlights the fundamental concepts of VQA, the challenges faced, different approaches and applications used for VQA tasks. A substantial portion of this review is devoted to investigating recent advancements in image feature extraction techniques. Findings: Most existing VQA research predominantly emphasizes the accurate matching of answers to given questions, often overlooking the necessity for a comprehensive representation of images. These models primarily rely on question content analysis while underemphasizing image understanding or sometimes neglect image examination entirely. There is also a tendency in multimodal systems to neglect or overemphasize one modality, notably the visual one, which challenges genuine multimodal integration. This article reveals that there is limited benchmarking for image feature extraction techniques. Evaluating the quality of extracted image features is crucial for VQA tasks. Novelty: While many VQA studies have primarily concentrated on the accuracy of answers to questions, this review emphasizes the importance of comprehensive image representation. The paper explores recent advances in Capsules Networks (CapsNets) and Vision Transformers (ViTs) as alternatives to traditional Convolutional Neural Networks (CNNs), for development of more effective image feature extraction techniques which can help to address the limitations of existing VQA models that focus primarily on question content analysis. https://www.indjst.org/

show abstract

Section: Transformer Based Approaches For Visual Question Answeringmentioning

confidence: 99%

Section: Vision Transformers For Visual Question Answeringmentioning

confidence: 99%

Section: Faster Rcnnmentioning

confidence: 99%

See 1 more Smart Citation

A Review of Recent Advances in Visual Question Answering: Capsule Networks and Vision Transformers in Focus

Prakash,

Devananda

2024

IJST

View full text Add to dashboard Cite

show abstract

“…It covers 12 diseases on 39 human parts. The medical datasets in 2022 are OVQA dataset [38], Diabetic Macular Edema (DME) dataset [28], EndoVis-18-VQA [39], and Cholec80-VQA [39].…”

Section: B Existing Vqa Datasetsmentioning

confidence: 99%

A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering

Al-Hadhrami,

Menai,

Al-Ahmadi

et al. 2023

IEEE Access

View full text Add to dashboard Cite

This paper comprehensively reviews medical VQA models, structures, and datasets, focusing on combining vision and language. Over 75 models and their statistical and SWOT (Strengths, Weaknesses, Opportunities, Threats) analyses were compared and analyzed. The study highlights whether the researchers in the general field influence those in the medical field. According to an analysis of text encoding techniques, LSTM is the approach that is utilized the most (42%), followed by non-text methods (14%) and BiLSTM (12%), whereas VGGNet (40%) and ResNet (22%) are the most often used vision methods, followed by Ensemble approaches (16%). Regarding fusion techniques, 14% of the models employed non-specific methods, while SAN (13%) and concatenation (10%) were frequently used. The study identifies LSTM-VGGNet and LSTM-ResNet combinations as the primary approaches in medical VQA, with 18% and 15% usage rates, respectively. The statistical analysis of medical VQA from 2018 to 2023 and individual yearly analyses reveals consistent preferences for LSTM and VGGNet, except in 2018 when ResNet was more commonly used. The SWOT analysis provides insights into the strengths and weaknesses of medical VQA research, highlighting areas for future exploration. These areas include addressing limited dataset sizes, enhancing question diversity, mitigating unimodal bias, exploring multi-modal datasets, leveraging external knowledge, incorporating multiple images, ensuring practical medical application integrity, improving model interpretation, and refining evaluation methods. This paper's findings contribute to understanding medical VQA and offer valuable guidance for future researchers aiming to make advancements in this field.

show abstract

SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery

Seenivasan,

Islam,

Kannan

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Surgical-VQA: Visual Question Answering in Surgical Scenes Using Transformer

Cited by 15 publications

References 22 publications

A Review of Recent Advances in Visual Question Answering: Capsule Networks and Vision Transformers in Focus

A Review of Recent Advances in Visual Question Answering: Capsule Networks and Vision Transformers in Focus

A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering

SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery

Contact Info

Product

Resources

About