No abstract

Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world -they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level visual concepts in the image that models ideally should understand to be able to answer the higher level question correctly. To address this, we first present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image, and use this to evaluate VQA models on their ability to identify the relevant sub-questions needed to answer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant subquestions higher than irrelevant questions for an pair. We show that SOrT improves model consistency by upto 6.5% points over existing baselines, while also improving visual grounding.

No abstract

Many machine learning methods operate by inverting a neural network at inference time, which has become a popular technique for solving inverse problems in computer vision, robotics, and graphics. However, these methods often involve gradient descent through a highly non-convex loss landscape, causing the optimization process to be unstable and slow. We introduce a method that learns a loss landscape where gradient descent is efficient, bringing massive improvement and acceleration to the inversion process. We demonstrate this advantage on a number of methods for both generative and discriminative tasks, including GAN inversion, adversarial defense, and 3D human pose reconstruction.Preprint. Under review.

Figure 1. Video Occlusions -Although the motorcycle (circled in yellow) becomes fully occluded in the video, we can still perform many visual recognition tasks, such as predicting its location, reconstructing its appearance, and classifying its semantic category. This paper introduces a video representation architecture that is able to learn to perform all of these occlusion reasoning tasks. We show example inputs and ground truths of the proposed dynamic scene completion framework.