Knowledge-Based Visual Question Generation

Xie, Jiayuan; Fang, Wenhao; Cai, Yi; Huang, Qingbao; Li, Qing

doi:10.1109/tcsvt.2022.3189242

Cited by 16 publications

(1 citation statement)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the rise of neural networks, VQG was formulated as an image-to-sequence problem, designing an image encoder followed by a decoder to generate questions in natural language (Ren, Kiros, and Zemel 2015;Mostafazadeh et al 2016;Li et al 2018;Patro et al 2018). However, these approaches often lead to poorly image-grounded and generic questions (Xie et al 2022;Krishna, Bernstein, and Fei-Fei 2019). To avoid generic questions, text-guided VQG has emerged, providing systems some guidance to obtain questions with specific properties.…”

Section: Related Workmentioning

confidence: 99%

ConVQG: Contrastive Visual Question Generation with Multimodal Guidance

Mi,

Montariol,

Castillo Navarro

et al. 2024

AAAI

View full text Add to dashboard Cite

Asking questions about visual environments is a crucial way for intelligent agents to understand rich multi-faceted scenes, raising the importance of Visual Question Generation (VQG) systems. Apart from being grounded to the image, existing VQG systems can use textual constraints, such as expected answers or knowledge triplets, to generate focused questions. These constraints allow VQG systems to specify the question content or leverage external commonsense knowledge that can not be obtained from the image content only. However, generating focused questions using textual constraints while enforcing a high relevance to the image content remains a challenge, as VQG systems often ignore one or both forms of grounding. In this work, we propose Contrastive Visual Question Generation (ConVQG), a method using a dual contrastive objective to discriminate questions generated using both modalities from those based on a single one. Experiments on both knowledge-aware and standard VQG benchmarks demonstrate that ConVQG outperforms the state-of-the-art methods and generates image-grounded, text-guided, and knowledge-rich questions. Our human evaluation results also show preference for ConVQG questions compared to non-contrastive baselines.

show abstract