ClipCap: CLIP Prefix for Image Captioning

Abstract

Image captioning is a fundamental task in visionlanguage understanding, where the model predicts a textual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. The recently proposed CLIP model contains rich semantic features which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we obtain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without additional annotations or pre-training, it efficiently generates meaningful captions for large-scale and diverse datasets. Surprisingly, our method works well even when only the mapping network is trained, while both CLIP and the language model remain frozen, allowing a lighter architecture with less trainable parameters. Through quantitative evaluation, we demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets, while it is simpler, faster, and lighter. Our code is available in https://github. com/rmokady/CLIP_prefix_caption.

ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

“…However, this ability is exactly the generative task DALL-E was trained to do, only in new domains. No previous computer vision work, as far as we can ascertain, has Method B@1 R@5 C-s B@1 R@5 C-s B@1 R@5 C-s B@1 R@5 C-s B@1 R@5 C-s CLIP-Prefix [49] 2. Comparison of our method and CLIP-Prefix baseline on our novel benchmark for visual relations.…”

“…1, we present our results for COCO's test set [42]. Two recent baselines that use CLIP's embedding are compared to: CLIP-Prefix [49] and CLIP-VL [61]. In CLIP-Prefix, the image is encoded using CLIP and the representation is transferred and plugged as a token into a fine-tuned GPT-2.…”

“…We compared our results with CLIP-Prefix [49] that encodes the image with CLIP's image encoder and uses it as an initial token for GPT-2. The method is fine-tuned based on COCO dataset.…”

“…Generated captions by our method and by the baseline methods for images from the MS-COCO [42] test-set. CP=CLIP-Prefix [49], CVL=CLIP-VL [61], VVL=VinVL [74].…”

“…14 (shown at the end of the document due to size), we present our results on 200 randomly-selected images along with baselines. For baselines, we use CLIPPrefix [49], CLIP-VL [61], and VinVL [74]. Our method generates original captions that…”