Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Abstract

We present a new dataset of image caption annotations, Conceptual Captions, which contains an order of magnitude more images than the MS-COCO dataset (Lin et al., 2014) and represents a wider variety of both images and image caption styles. We achieve this by extracting and filtering image caption annotations from billions of webpages. We also present quantitative evaluations of a number of image captioning models and show that a model architecture based on Inception-ResNet-v2 (Szegedy et al., 2016) for image-feature extraction and Transformer (Vaswani et al., 2017) for sequence modeling achieves the best performance when trained on the Conceptual Captions dataset.

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

“…Second, we report the results in the unsupervised setting with independent image and language sources. We experiment with Flickr30k Images [49] paired with COCO captions and COCO images paired with Google's Conceptual Captions dataset (GCC) [52]. Finally, we show qualitative results for image descriptions with varying text sources.…”

Large-Scale Representation Learning from Visually Grounded Untranscribed Speech

“…Image captioning datasets have ignited a great deal of research at the intersection of the computer vision and natural language processing communities (Lin et al, 2014;Vinyals et al, 2015;Bernardi et al, 2016;Anderson et al, 2018). Getting annotators to provide captions works well with crowd computing, but Sharma et al (2018) exploit incidental supervision for this task to obtain greater scale with their Conceptual Captions dataset. It contains 3.3 million pairs of image and textual captions, where pairs are extracted from HTML web pages using the alt-text field of images as a starting point for their descriptions.…”

“…There are many sequential filtering steps for improving the quality of the captions-see Sharma et al (2018) for a thorough description. As quality control, a random sample of 4K conceptual captions were rated by human annotators, and 90.3% were judged "good" by at least 2 out of 3 raters.…”

Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering

“…Our Objective The main goal is to compare two approaches in using bottom-up signals: 1) FRCNN: use the default visual features from the Faster R-CNN detector; 2) Ultra: use bounding boxes from the Faster R-CNN detector, then fea-4 Image Captioning Dataset We use the Conceptual Captions (CC) dataset (Sharma et al, 2018), consisting of 3.3 million training and 15,000 validation images/caption pairs. Another 12,000 image/caption pairs comprise the hidden test set.…”

“…Model We adopt the encoder-decoder model from (Sharma et al, 2018), whose basic building block is a Transformer Network (Vaswani et al, 2017). To convert multi-modal inputs to a sequence of encoder feature vectors, we use up to three types of image features: L : Label embeddings, obtained by embedding predicted object semantic labels from Google Cloud Vision APIs 3 into a 512D feature vector.…”