Focal Visual-Text Attention for Memex Question Answering

Liang, Junwei; Jiang, Lu; Cao, Liangliang; Kalantidis, Yannis; Li, Lijia; Hauptmann, Alexander G.

doi:10.1109/tpami.2018.2890628

Cited by 71 publications

(25 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from answering questions, J. Liang et al [86], have reported evidential image and text snippets to support the reasoning for the answer. The proposed attention-based neural network, Focal Visual-Text Attention network (FVTA) takes into consideration both visual and text sequence information.…”

Section: ) Encoder-decoder Based Methodsmentioning

confidence: 99%

Video Question-Answering Techniques, Benchmark Datasets and Evaluation Metrics Leveraging Video Captioning: A Comprehensive Survey

Khurana¹,

Deshpande

2021

IEEE Access

View full text Add to dashboard Cite

While describing visual data is a trivial task for humans, it is an intricate task for a computer. This is even more challenging if the visual data is a video. Comprehending a video and describing it is called Video Captioning. This involves understanding the semantics of a video and then generating humanlike descriptions of the video. It requires the collaboration of both research communities of computer vision and natural language processing. The captions generated by video captioning can be further utilized for video retrieval, summarization, question-answering, etc. Video Question-Answering (video-QA) involves querying the system to obtain an answer in response. This paper presents a brief survey of the video captioning techniques and a comprehensive review of existing techniques, datasets, and evaluation metrics for the task of video-QA. Video-QA techniques rely on the attention mechanism to generate relevant results. The presented survey shows that recent works on Memory Networks, Generative Adversarial Networks, and Reinforced Decoders, have the capability to handle the complexities and challenges of video-QA. Additionally, the graph-based methods, although less explored, give very promising results. In this article, we have discussed the emerging research directions and various application areas of video-QA.

show abstract

Section: ) Encoder-decoder Based Methodsmentioning

confidence: 99%

Video Question-Answering Techniques, Benchmark Datasets and Evaluation Metrics Leveraging Video Captioning: A Comprehensive Survey

Khurana¹,

Deshpande

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Multimodal Feature Composition. Another related area is multimodal feature composition which has been studied more extensively in other problems such as visual question answering [8,30,44,53], visual reasoning [28,60], image-to-image translation [34,83], etc. Specifically, our method is related to feature-wise modulation, a technique to modulate the features of one source by referencing those from the other.…”

Section: Related Workmentioning

confidence: 99%

Text as Neural Operator:Image Manipulation by Text Instruction

Zhang

Tseng

Jiang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

show abstract

“…The VQA task (Agrawal et al, 2015) requires answering a freeform natural language question about visual content in an image. Previous work has shown that models often do well on the task by exploiting language and dataset biases (Agrawal et al, 2017;Zhang et al, 2015;Ramakrishnan et al, 2018;Guo et al, 2019;Manjunatha et al, 2018 (Selvaraju et al, 2019a,b;Qiao et al, 2017;Liang et al, 2019), the multi-modal task of VQA has a language component which cannot always be explained visually, i.e., visual regions can be insufficient to express underlying concepts (Goyal et al, 2016;Hu et al, 2017). Park et al (2018) and Wu and Mooney (2019) generate textual justifications through datasets curated with human explanations.…”

Section: Related Workmentioning

confidence: 99%

SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency

Dharur¹,

Tendulkar²,

Batra³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world -they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level visual concepts in the image that models ideally should understand to be able to answer the reasoning question correctly. To address this, we first present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image, and use this to evaluate VQA models on their ability to identify the relevant sub-questions needed to answer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant subquestions higher than irrelevant questions for an pair. We show that SOrT improves model consistency by up to 6.5% points over existing approaches, while also improving visual grounding and robustness to rephrasings of questions.

show abstract

Focal Visual-Text Attention for Memex Question Answering

Cited by 71 publications

References 50 publications

Video Question-Answering Techniques, Benchmark Datasets and Evaluation Metrics Leveraging Video Captioning: A Comprehensive Survey

Video Question-Answering Techniques, Benchmark Datasets and Evaluation Metrics Leveraging Video Captioning: A Comprehensive Survey

Text as Neural Operator:Image Manipulation by Text Instruction

SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency

Contact Info

Product

Resources

About