Multi-Level Multimodal Common Semantic Space for Image-Phrase Grounding

Akbari, Hassan; Karaman, Svebor; Bhargava, Surabhi; Chen, Brian; Vondrick, Carl; Chang, Shih-Fu

doi:10.1109/cvpr.2019.01276

Cited by 65 publications

(66 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For every image, we select the top 30 RoIs based on Faster-RCNN's class detection score (after non-maximal suppre- sion and thresholding). 2 We then use RoIAlign [17] to extract features (d v = 2048-d) for each of these RoIs using a ResNet-152 model pre-trained on ImageNet [18].…”

Section: Implementation Detailsmentioning

confidence: 99%

“…For phrase localization, we outperform the previous state-of-the-art [13], which uses a variant of the Global method with a novel spatial pooling step, by 4.9% based on the P ointIt% metric on VG. On Flickr30k Entities, we out-perform prior stateof-the-art [2] with much simpler encoders (ResNet+bi-GRU v/s PNASNet+ElMo). For the caption-to-image retrieval task, we also achieve state-of-the-art performance (R@1 of 49.7 vs. 48.6 by [25]) on Flickr30k dataset and get competitive results relative to state-of-the-art (R@1 of 56.6 vs. 58.8 by [25]) on COCO dataset for the downstream C2I task.…”

Section: Comparison With State-of-the-artmentioning

confidence: 99%

See 1 more Smart Citation

Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment

Datta

Sikka

Roy

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

We address the problem of grounding free-form textual phrases by using weak supervision from image-caption pairs. We propose a novel end-to-end model that uses caption-toimage retrieval as a "downstream" task to guide the process of phrase localization. Our method, as a first step, infers the latent correspondences between regions-of-interest (RoIs) and phrases in the caption and creates a discriminative image representation using these matched RoIs. In the subsequent step, this learned representation is aligned with the caption. Our key contribution lies in building this "captionconditioned" image encoding which tightly couples both the tasks and allows the weak supervision to effectively guide visual grounding. We provide extensive empirical and qualitative analysis to investigate the different components of our proposed model and compare it with competitive baselines. For phrase localization, we report improvements of 4.9% and 1.3% (absolute) over prior state-of-the-art on the VisualGenome and Flickr30k Entities datasets. We also report results that are at par with the state-of-the-art on the downstream caption-to-image retrieval task on COCO and Flickr30k datasets.Recent works [20,21] have shown evidence that operating under such a paradigm helps boost performance for imagecaption matching. Generally, these models consist of two stages: (1) a local matching module that infers the latent region-phrase correspondences to generate local matching information, and (2) a global matching module that uses this information to perform image-caption matching. This setup allows phrase grounding to act as an intermediate and a prerequisite task for image-caption matching. It is important to note that the primary objective of such works has been on image-caption matching and not phrase grounding.An artifact of training under such a paradigm is the amplification of correlations between selective regions and phrases."Young girl holding a kitten" by Gennadiy Kolodkin is licensed under CC BY-NC-ND 2.0.

show abstract

Section: Implementation Detailsmentioning

confidence: 99%

Section: Comparison With State-of-the-artmentioning

confidence: 99%

Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment

Datta

Sikka

Roy

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…Inspired by the work of [2] and [12], we take the approach of using multiple autoencoders [53] that each learn a common feature space for multiple modalities. Assuming that the common feature space has a dimensionality of , the mapping function in the encoder for each autoencoder ( ) outputs a vector with dimensionality of .…”

Section: Mapping Inputs To a Common Feature Spacementioning

confidence: 99%

“…(2)ĉ E (1) E (2) D (1) D (2) Figure 2: An illustration of the mapping module with two modalities. Figure 2 provides an illustration with an example of two modalities mapped into a common feature space and then reconstructed, based on two autoencoders.…”

Section: ( )mentioning

confidence: 99%

Multimodal Data Fusion based on the Global Workspace Theory

Bao

Fountas

Olugbade

et al. 2020

Proceedings of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

We propose a novel neural network architecture, named the Global Workspace Network (GWN), which addresses the challenge of dynamic and unspecified uncertainties in multimodal data fusion. Our GWN is a model of attention across modalities and evolving through time, and is inspired by the well-established Global Workspace Theory from the field of cognitive science. The GWN achieved average F1 score of 0.92 for discrimination between pain patients and healthy participants and average F1 score = 0.75 for further classification of three pain levels for a patient, both based on the multimodal EmoPain dataset captured from people with chronic pain and healthy people performing different types of exercise movements in unconstrained settings. In these tasks, the GWN significantly outperforms the typical fusion approach of merging by concatenation. We further provide extensive analysis of the behaviour of the GWN and its ability to address uncertainties (hidden noise) in multimodal data. CCS CONCEPTS • Computing methodologies → Machine learning algorithms; Neural networks.

show abstract

“…Phrase grounding. Most existing works, if not all, can be categorized into two types: one is attention-based [2,4] and embeddingbased [10,29]. The former type treats the input text query as key to generate a pixel-level or coarser attention map and use this map to further generate bounding boxes.…”

Section: Related Workmentioning

confidence: 99%

Cross-Modal Omni Interaction Modeling for Phrase Grounding

Hui

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Phrase grounding aims to localize the objects described by phrases in a natural language specification. Previous works model the interaction of inputs from text modality and visual modality only in the intra-modal global level and consequently lacks the ability to capture the precise and complete context information. In this paper, we propose a novel Cross-Modal Omni Interaction network (COI Net) composed of a neighboring interaction module, a global interaction module, a cross-modal interaction module and a multilevel alignment module. Our approach formulates the complex spatial and semantic relationship among image regions and phrases through multi-level multi-modal interaction. We capture the local relationship using the interaction among neighboring regions and then collect the global context through the interaction among all regions using a transformer encoder. We further use a co-attention module to apply the interaction between two modalities to gather the crossmodal context for all image regions and phrases. In addition to the omni interaction modeling, we also leverage a straightforward yet effective multilevel alignment regularization to formulate the dependencies among all grounding decisions. We extensively validate the effectiveness of our model. Experiments show that our approach outperforms existing state-of-the-art methods by large margins on two popular datasets in terms of accuracy: 6.15% on Flickr30K Entities (71.36% increased to 77.51%) and 21.25% on ReferItGame (44.91% * Equal contribution.

show abstract

Multi-Level Multimodal Common Semantic Space for Image-Phrase Grounding

Cited by 65 publications

References 51 publications

Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment

Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment

Multimodal Data Fusion based on the Global Workspace Theory

Cross-Modal Omni Interaction Modeling for Phrase Grounding

Contact Info

Product

Resources

About