Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA

Abstract

Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. Contrast sets quantify this phenomenon by perturbing test samples in a minimal way such that the output label is modified. While most contrast sets were created manually, requiring intensive annotation effort, we present a novel method which leverages rich semantic input representation to automatically generate contrast sets for the visual question answering task. Our method computes the answer of perturbed questions, thus vastly reducing annotation cost and enabling thorough evaluation of models' performance on various semantic aspects (e.g., spatial or relational reasoning). We demonstrate the effectiveness of our approach on the popular GQA dataset (Hudson and Manning, 2019) and its semantic scene graph image representation. We find that, despite GQA's compositionality and carefully balanced label distribution, two strong models drop 13-17% in accuracy on our automatically-constructed contrast set compared to the original validation set. Finally, we show that our method can be applied to the training set to mitigate the degradation in performance, opening the door to more robust models. 1

CARETS: A Consistency And Robustness Evaluative Test Suite for VQA

“…6 Results specific concepts, corroborating the findings of (Bitton et al, 2021). Interestingly, the best performing model (LXMERT) is not always the most consistent.…”

“…Some recent work has sought to evaluate models using consistency and other metrics (Hudson and Manning, 2019;Shah et al, 2019;Ribeiro et al, 2020a;Selvaraju et al, 2020;Bitton et al, 2021). These evaluations often evaluate consistency through question entailment and implication, or simply contrasting examples in the case of (Bitton et al, 2021).…”

“…Some recent work has sought to evaluate models using consistency and other metrics (Hudson and Manning, 2019;Shah et al, 2019;Ribeiro et al, 2020a;Selvaraju et al, 2020;Bitton et al, 2021). These evaluations often evaluate consistency through question entailment and implication, or simply contrasting examples in the case of (Bitton et al, 2021). While we consider such methods important for evaluating model comprehension, they often combine question types and capabilities, changing the kind of expected answer, or evaluating consistency on a tree or set of entailed questions.…”

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition

“…We generate perturbations at the level of the underlying reasoning process, in the context of QA. Last, Bitton et al (2021) used scene graphs to generate examples for visual QA. However, they assumed the existence of gold scene graph at the input.…”

“…Recently, methods for automatic generation of contrast sets were proposed. However, current methods are restricted to shallow surface perturbations (Mille et al, 2021;, specific reasoning skills , or rely on expensive annotations (Bitton et al, 2021). Thus, automatic generation of examples that test high-level reasoning abilities of models and their robustness to fine semantic distinctions remains an open challenge.…”