Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections

Gong, Yunchao; Wang, Liwei; Hodosh, Micah; Hockenmaier, Julia; Lazebnik, Svetlana

doi:10.1007/978-3-319-10593-2_35

Cited by 240 publications

(168 citation statements)

References 27 publications

Supporting

Mentioning

167

Contrasting

Order By: Relevance

“…In addition, all 3, 361 image-text pairs from the BBC News Corpora and 2, 999 image-text pairs from the SimpleWiki dataset have been included. From this randomly shuffled corpus, samples have been selected to generate a disjoint split of 190, 202 training and 6, 270 validation samples 8 . The image encoding network has been initialized with weights of a pre-trained InceptionV3 model.…”

Section: Methodsmentioning

confidence: 99%

Estimating the information gap between textual and visual representations

Henning

Ewerth

2017

Int J Multimed Info Retr

View full text Add to dashboard Cite

Photos, drawings, figures, etc. supplement textual information in various kinds of media, for example, in web news or scientific publications. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general illustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be described and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe cross-modal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general illustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be described and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe crossmodal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach.

show abstract

Section: Methodsmentioning

confidence: 99%

Estimating the information gap between textual and visual representations

Henning

Ewerth

2017

Int J Multimed Info Retr

View full text Add to dashboard Cite

show abstract

“…dataset (Young et al 2014), a popular benchmark for caption generation and retrieval that has been used, among others, by Chen and Zitnick (2015); Donahue et al (2015); Fang et al (2015); Gong et al (2014b); Karpathy et al (2014); Karpathy and Fei-Fei (2015); Kiros et al (2014); Klein et al (2014); Lebret et al (2015); Mao et al (2015); Vinyals et al (2015); Xu et al (2015). Flickr30k contains 31,783 images focusing mainly on people and animals, and 158,915 English captions (five per image).…”

Section: Figmentioning

confidence: 99%

“…This approach learns an embedding of region and phrase features to a shared latent space and uses distance in that space to retrieve image regions given a phrase. While there have been several neural network-based approaches for learning such embeddings (Karpathy and Fei-Fei 2015;Kiros et al 2014;Mao et al 2015), using state-of-the-art text and image features with Canonical Correlation Analysis (CCA) (Hotelling 1936) continues to produce remarkable results (Gong et al 2014b;Klein et al 2014;Lev et al 2016), and is also much faster to train than a neural network. Given two sets of matching features from different views (in our case, image and text features), CCA finds linear projections of both views into a joint space of common dimensionality in which the correlation between the views is maximized.…”

Section: Region-phrase Modelmentioning

confidence: 99%

“…Our split is the same as in Gong et al (2014b). We evaluate localization performance by treating the phrase as the query to retrieve the proposals from the input image and report Recall@K (K = 1, 5, 10), or the percentage of queries for which a correct match has rank of at most K (we deem a region to be a correct match if it has IOU ≥ 0.5 with the ground truth bounding box for that phrase).…”

Section: Evaluation Protocolmentioning

confidence: 99%

See 1 more Smart Citation

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Plummer

Wang

Cervantes

et al. 2015

2015 IEEE International Conference on Computer Vision (ICCV)

Self Cite

955

107

View full text Add to dashboard Cite

The Flickr30k dataset has become a standard benchmark for sentence-based image description. This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains, linking mentions of the same entities across different captions for the same image, and associating them with 276k manually annotated bounding boxes. Such annotations are essential for continued progress in automatic image description and grounded language understanding. They enable us to define a new benchmark for localization of textual entity mentions in an image. We present a strong baseline for this task that combines an image-text embedding, detectors for common objects, a color classifier, and a bias towards selecting larger objects. While our baseline rivals in accuracy more complex state-of-the-art models, we show that its gains cannot be easily parlayed into improvements on such tasks as image-sentence retrieval, thus underlining the limitations of current methods and the need for further research.

show abstract

“…Our corpus has several unique properties to complement existing corpora. As explored in a very recent work of (Gong et al, 2014), we expect that it is possible to combine crowd-sourced and web-harvested datasets and achieve the best of both worlds.…”

Section: Related Workmentioning

confidence: 99%

Déjà Image-Captions: A Corpus of Expressive Descriptions in Repetition

Chen¹,

Kuznetsova²,

Warren³

et al. 2015

Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

We present a new approach to harvesting a large-scale, high quality image-caption corpus that makes a better use of already existing web data with no additional human efforts. The key idea is to focus on Déjà Image-Captions: naturally existing image descriptions that are repeated almost verbatim -by more than one individual for different images. The resulting corpus provides association structure between 4 million images with 180K unique captions, capturing a rich spectrum of everyday narratives including figurative and pragmatic language. Exploring the use of the new corpus, we also present new conceptual tasks of visually situated paraphrasing, creative image captioning, and creative visual paraphrasing.

show abstract

Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections

Cited by 240 publications

References 27 publications

Estimating the information gap between textual and visual representations

Estimating the information gap between textual and visual representations

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Déjà Image-Captions: A Corpus of Expressive Descriptions in Repetition

Contact Info

Product

Resources

About