CLEVR3D: Compositional Language and Elementary Visual Reasoning for Question Answering in 3D Real-World Scenes

Abstract

3D scene understanding is a relatively emerging research field. In this paper, we introduce the Visual Question Answering task in 3D real-world scenes (VQA-3D), which aims to answer all possible questions given a 3D scene. To tackle this problem, the first VQA-3D dataset, namely CLEVR3D, is proposed, which contains 60K questions in 1,129 real-world scenes. Specifically, we develop a question engine leveraging 3D scene graph structures to generate diverse reasoning questions, covering the questions of objects' attributes (i.e., size, color, and material) and their spatial relationships. Built upon this dataset, we further design the first VQA-3D baseline model, TransVQA3D. The TransVQA3D model adopts well-designed Transformer ar-chitectures to achieve superior VQA-3D performance, compared with the pure language baseline and previous 3D reasoning methods directly applied to 3D scenarios. Experimental results verify that taking VQA-3D as an auxiliary task can boost the performance of 3D scene understanding, including scene graph analysis for the node-wise classification and whole-graph recognition.

Multi-modal assisted knowledge distillation for 3D question answering

“…is the main problem of 3D-QA. Recently, some 3D-QA datasets [1,2,8,9] have emerged at the same time. We choose ScanQA [2] as our research object since the other two are not open source yet.…”

Image to English translation and comprehension: INT2-VQA method based on inter-modality and intra-modality collaborations

“…Traditional 3D scenario understanding tasks mostly focus on individual targets and overlook target relations. Literature [ 43 ] introduced visual question answering in real 3D scenarios, aiming to answer all possible questions given a 3D scenario. They designed TransVQA3D, which first uses a cross-modal Transformer to fuse question and target characteristics.…”