Intra-Modality Feature Interaction Using Self-attention for Visual Question Answering

Shao, Huan; Xu, Yuehang; Yang, Jianyu; Liu, Chunping

doi:10.1007/978-3-030-36802-9_24

Cited by 4 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, some approaches [49], [57], [58], [59], [60] incorporated object detectors to enhance the extraction of visual features. Attention mechanisms [49], [61], [62] and graph neural networks [44], [63] have gained increasing popularity in recent years due to their ability to capture fine-grained visual details and contextual relationships. Notably, methods incorporating VLP models, such as ConZIC [64], have demonstrated efficient performance in image captioning.…”

Section: A Image Captioningmentioning

confidence: 99%

A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes

Lin

Wang

et al. 2024

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

Section: A Image Captioningmentioning

confidence: 99%

A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes

Lin

Wang

et al. 2024

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Researchers have successfully employed the visual attention method in the VQA task in the past few years. However, the traditional attention methods are extended to different directions, such as region-based attention methods, [1][2][3] object-based attention, [4][5][6] and semantic concept-based attention methods. [7][8][9] However, in all these methods, particular visual features such as region, object, or semantic concept features are considered to construct the attention mechanism, which is insufficient for feature representation.…”

Section: Introductionmentioning

confidence: 99%

Enhancing visual question answering with a two‐way co‐attention mechanism and integrated multimodal features

Agrawal,

Jalal,

Sharma

2023

Computational Intelligence

View full text Add to dashboard Cite

In Visual question answering (VQA), a natural language answer is generated for a given image and a question related to that image. There is a significant growth in the VQA task by applying an efficient attention mechanism. However, current VQA models use region features or object features that are not adequate to improve the accuracy of generated answers. To deal with this issue, we have used a Two‐way Co‐Attention Mechanism (TCAM), which is capable enough to fuse different visual features (region, object, and concept) from diverse perspectives. These diverse features lead to different sets of answers, and also, there is an inherent relationship between these visual features. We have developed a powerful attention mechanism that uses these two critical aspects by using both bottom‐up and top‐down TCAM to extract discriminative feature information. We have proposed a Collective Feature Integration Module (CFIM) to combine multimodal attention features and thus capture the valuable information from these visual features by employing a TCAM. Further, we have formulated a Vertical CFIM for fusing the features belonging to the same class and a Horizontal CFIM for combining the features belonging to different types, thus balancing the influence of top‐down and bottom‐up co‐attention. The experiments are conducted on two significant datasets, VQA 1.0 and VQA 2.0. On VQA 1.0, the overall accuracy of our proposed method is 71.23 on the test‐dev set and 71.94 on the test‐std set. On VQA 2.0, the overall accuracy of our proposed method is 75.89 on the test‐dev set and 76.32 on the test‐std set. The above overall accuracy clearly reflecting the superiority of our proposed TCAM based approach over the existing methods.

show abstract

Integrating Historical States and Co-attention Mechanism for Visual Dialog

Jiang

Liu

2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Intra-Modality Feature Interaction Using Self-attention for Visual Question Answering

Cited by 4 publications

References 16 publications

A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes

A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes

Enhancing visual question answering with a two‐way co‐attention mechanism and integrated multimodal features

Integrating Historical States and Co-attention Mechanism for Visual Dialog

Contact Info

Product

Resources

About