Improving Generative Visual Dialog by Answering Diverse Questions

Abstract

Prior work on training generative Visual Dialog models with reinforcement learning (Das et al., 2017b) has explored a Q-BOT-A-BOT image-guessing game and shown that this 'self-talk' approach can lead to improved performance at the downstream dialogconditioned image-guessing task. However, this improvement saturates and starts degrading after a few rounds of interaction, and does not lead to a better Visual Dialog model. We find that this is due in part to repeated interactions between Q-BOT and A-BOT during selftalk, which are not informative with respect to the image. To improve this, we devise a simple auxiliary objective that incentivizes Q-BOT to ask diverse questions, thus reducing repetitions and in turn enabling A-BOT to explore a larger state space during RL i.e. be exposed to more visual concepts to talk about, and varied questions to answer. We evaluate our approach via a host of automatic metrics and human studies, and demonstrate that it leads to better dialog, i.e. dialog that is more diverse (i.e. less repetitive), consistent (i.e. has fewer conflicting exchanges), fluent (i.e. more human-like), and detailed, while still being comparably image-relevant as prior work and ablations. Our code is publicly available at github.com/vmurahari3/visdial-diversity. ReferencesSamy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS.

On the role of effective and referring questions in GuessWhat?!

“…We agree with (Thomason et al, 2019) that incremental evaluation metrics such as ours should look further back into the dialogue history. We believe that language and vision systems should also be evaluated on aspects such as grammatically, truthfulness, diversity and other aspects as done in previous work (Lee et al, 2018;Ray et al, 2019;Xie et al, 2020;Murahari et al, 2019). In this paper we focus on whether a question is effective and referential considering the dialogue history and the visual context.…”

They Are Not All Alike: Answering Different Spatial Questions Requires Different Grounding Strategies

“…other, which we call the Oracle, answers. For Vis-Dial most of the work focused on the answerer, but in-depth evaluation has been carried out on the questioner too (eg., Murahari et al (2019); Testoni et al (2019)). For GuessWhat?…”

“I’ve Seen Things You People Wouldn’t Believe”: Hallucinating Entities in GuessWhat?!

“…because of the simplicity of its dialogue structure (polar question-answer pairs). Recent work in the literature highlights the inability of the accuracy in the guessing task to serve as a good proxy of the quality of the underlying dialogues, with a particular focus on surface-level features such as the presence of repetitions Murahari et al, 2019;Testoni et al, 2019). We extend this claim by looking at hallucination, an under-studied but crucial issue in Visual Dialogues.…”