Weakly Supervised Attention Learning for Textual Phrases Grounding

Abstract

Grounding textual phrases in visual content is a meaningful yet challenging problem with various potential applications such as image-text inference or text-driven multimedia interaction. Most of the current existing methods adopt the supervised learning mechanism which requires groundtruth at pixel level during training. However, fine-grained level ground-truth annotation is quite time-consuming and severely narrows the scope for more general applications. In this extended abstract, we explore methods to localize flexibly image regions from the top-down signal (in a form of one-hot label or natural languages) with a weakly supervised attention learning mechanism. In our model, two types of modules are utilized: a backbone module for visual feature capturing, and an attentive module generating maps based on regularized bilinear pooling. We construct the model in an end-to-end fashion which is trained by encouraging the spatial attentive map to shift and focus on the region that consists of the best matched visual features with the top-down signal. We demonstrate the preliminary yet promising results on a testbed that is synthesized with multi-label MNIST data.

Weak Supervision and Referring Attention for Temporal-Textual Association Learning

“…Association Learning across Vision and Language is core and tie of a wide range of tasks across vision and language domains, e.g., textual grounding [38], referring expression comprehension [31] or object retrieval using language [16]. Recent works focus on leveraging the imagelevel annotations (as weak supervision) [8,9] or unsupervised method [64] to learn the association across language descriptions and objects. Proceeding from this, there arise works on using uncurated captions to learn temporal associations across video segments and texts [27,51].…”

Modularized Textual Grounding for Counterfactual Resilience

“…State-of-the-art textual grounding methods [69,34,60,58,66,47,16] are based on deep neural networks and relying on large-scale training data with manual annotations for the object bounding box and relationship between phrases and figures/objects. This setup largely limits their broad applications as such strong supervision is expensive to obtain, and they also lack interpretability and resilience to counterfactual cases which do not appear in training.…”