Visual Question Generation for Class Acquisition of Unknown Objects

Abstract

Traditional image recognition methods only consider objects belonging to already learned classes. However, since training a recognition model with every object class in the world is unfeasible, a way of getting information on unknown objects (i.e., objects whose class has not been learned) is necessary. A way for an image recognition system to learn new classes could be asking a human about objects that are unknown. In this paper, we propose a method for generating questions about unknown objects in an image, as means to get information about classes that have not been learned. Our method consists of a module for proposing objects, a module for identifying unknown objects, and a module for generating questions about unknown objects. The experimental results via human evaluation show that our method can successfully get information about unknown objects in an image dataset. Our code and dataset are available at https://github.com/mil-tokyo/vqg-unknown

Unsupervised Keyword Extraction for Full-Sentence VQA

“…In future work, the extracted keywords will be utilized in other tasks, such as VQA, object classification, or object detection. This work could also be combined with recent works on VQG (Uehara et al, 2018;Shen et al, 2019). In these works, the system generates questions to acquire information from humans.…”

“…Thus, to train image recognition models, it is important for real-world intelligent systems to actively acquire information. One promising approach to acquire information on the fly is learning by asking, i.e., generating questions to humans about unknown objects, and consequently learning new knowledge from the human response (Misra et al, 2018;Uehara et al, 2018;Shen et al, 2019). This implies that if we can build a Visual Question Answering (VQA) system (Antol et al, 2015) that functions in the real Figure 1: Example of the proposed task -keyword extraction from full-sentence VQA.…”

Learning to Caption Images Through a Lifetime by Asking Questions

“…However, questions are limited to templates, and training is done in synthetic environments with a limited set of objects and relationships. [28] uses questions to explore new object classes for image classification. However, [28] does not retrain their classifier.…”

“…[28] uses questions to explore new object classes for image classification. However, [28] does not retrain their classifier. Our work differs from [33,28] by proposing a way for the agent to learn in a lifetime setting.…”

“…However, [28] does not retrain their classifier. Our work differs from [33,28] by proposing a way for the agent to learn in a lifetime setting.…”