Grounding Visual Explanations

Hendricks, Lisa Anne; Hu, Ronghang; Darrell, Trevor; Akata, Zeynep

doi:10.1007/978-3-030-01216-8_17

Cited by 171 publications

(127 citation statements)

References 40 publications

Supporting

Mentioning

126

Contrasting

Order By: Relevance

“…Task-specific caption generation While most image captioning works focus on a generic task of obtaining image relevant descriptions [3,12,58], some recent works explore pragmatic or "task-specific" captions. Some focus on generating textual explanations for deep models' predictions [19,20,45]. Others aim to generate a discriminative caption for an image or image region, to disambiguate it from a distractor [4,10,40,39,56,62].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Robust Change Captioning

Park

Darrell

Rohrbach

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

119

View full text Add to dashboard Cite

Describing what has changed in a scene can be useful to a user, but only if generated text focuses on what is semantically relevant. It is thus important to distinguish distractors (e.g. a viewpoint change) from relevant changes (e.g. an object has moved). We present a novel Dual Dynamic Attention Model (DUDA) to perform robust Change Captioning. Our model learns to distinguish distractors from semantic changes, localize the changes via Dual Attention over "before" and "after" images, and accurately describe them in natural language via Dynamic Speaker, by adaptively focusing on the necessary visual inputs (e.g. "before" or "after" image). To study the problem in depth, we collect a CLEVR-Change dataset, built off the CLEVR engine, with 5 types of scene changes. We benchmark a number of baselines on our dataset, and systematically study different change types and robustness to distractors. We show the superiority of our DUDA model in terms of both change captioning and localization. We also show that our approach is general, obtaining state-of-the-art results on the recent realistic Spot-the-Diff dataset which has no distractors.

show abstract

Section: Related Workmentioning

confidence: 99%

“…They can be especially useful, when tailored to a specific task or objective, such as e.g. explaining the model's predictions [20,45] or generating non-ambiguous referring expressions for specific image regions [40,62].…”

Section: Introductionmentioning

confidence: 99%

Robust Change Captioning

Park

Darrell

Rohrbach

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

119

View full text Add to dashboard Cite

show abstract

“…In the counterfactual visual explanation [22] work, a patch based editing of input image is optimize in order to satisfy the intended changes in the prediction. In the ground visual explanation [23] work, text based explanation are generated to provide counterfactual explanation for image classification task. Beside the causal interpretation methods, as demonstrated in [9], examining the relationship between the trained model and training dataset can also help interpret model behavior.…”

Section: Related Workmentioning

confidence: 99%

Generative Counterfactual Introspection for Explainable Deep Learning

Liu

Kailkhura

Loveland

et al. 2019

2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP)

View full text Add to dashboard Cite

In this work, we propose an introspection technique for deep neural networks that relies on a generative model to instigate salient editing of the input image for model interpretation. Such modification provides the fundamental interventional operation that allows us to obtain answers to counterfactual inquiries, i.e., what meaningful change can be made to the input image in order to alter the prediction. We demonstrate how to reveal interesting properties of the given classifiers by utilizing the proposed introspection approach on both the MNIST and the CelebA dataset.

show abstract

“…Few works have treated multimodality for explanation [21,2], which is visual and linguistic. Although they provided visual information by referring to a part of the target samples, we explore the method to utilize other examples for explanation.…”

Section: Related Workmentioning

confidence: 99%

Learning to Explain With Complemental Examples

Kanehira

Harada

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

This paper addresses the generation of explanations with visual examples. Given an input sample, we build a system that not only classifies it to a specific category, but also outputs linguistic explanations and a set of visual examples that render the decision interpretable. Focusing especially on the complementarity of the multimodal information, i.e., linguistic and visual examples, we attempt to achieve it by maximizing the interaction information, which provides a natural definition of complementarity from an information theoretical viewpoint. We propose a novel framework to generate complemental explanations, on which the joint distribution of the variables to explain, and those to be explained is parameterized by three different neural networks: predictor, linguistic explainer, and example selector. Explanation models are trained collaboratively to maximize the interaction information to ensure the generated explanation are complemental to each other for the target. The results of experiments conducted on several datasets demonstrate the effectiveness of the proposed method.

show abstract

Grounding Visual Explanations

Cited by 171 publications

References 40 publications

Robust Change Captioning

Robust Change Captioning

Generative Counterfactual Introspection for Explainable Deep Learning

Learning to Explain With Complemental Examples

Contact Info

Product

Resources

About