Fashion-Oriented Image Captioning with External Knowledge Retrieval and Fully Attentive Gates

Abstract

Research related to fashion and e-commerce domains is gaining attention in computer vision and multimedia communities. Following this trend, this article tackles the task of generating fine-grained and accurate natural language descriptions of fashion items, a recently-proposed and under-explored challenge that is still far from being solved. To overcome the limitations of previous approaches, a transformer-based captioning model was designed with the integration of external textual memory that could be accessed through k-nearest neighbor (kNN) searches. From an architectural point of view, the proposed transformer model can read and retrieve items from the external memory through cross-attention operations, and tune the flow of information coming from the external memory thanks to a novel fully attentive gate. Experimental analyses were carried out on the fashion captioning dataset (FACAD) for fashion image captioning, which contains more than 130k fine-grained descriptions, validating the effectiveness of the proposed approach and the proposed architectural strategies in comparison with carefully designed baselines and state-of-the-art approaches. The presented method constantly outperforms all compared approaches, demonstrating its effectiveness for fashion image captioning.

Arabic Captioning for Images of Clothing Using Deep Learning

“…This demonstrates the success of training the model utilizing image attributes in addition to the image features and linguistic features and represents an improvement of more than 30 BLEU points over the model that does not use attributes and other related models. For example, our model outperforms [22] with more than 40 BLEU points and [23] with more than 60 BLEU points.…”

“…For the second category, it considers the models developed for the fashion domain which includes [20] that explored training the model using diversity datasets [1] which proposed a smaller 5-layer Convolutional Neural Network (CNN-5) to extract image features, Semantic Rewards guided Fashion Captioning (SRFC) [21]. It also includes [22] which explored taking the semantic attributes from the users, and [23] which developed a transformer-based model. As can be observed, according to BLEU, the proposed approach performs better than all of the methods that were compared.…”

“…Moratelli et al [23] presented a transformer-based captioning model that included external text memory that could be retrieved using k-nearest neighbor (KNN) searches. They encoded both the images and the text using a vision and textual transformer respectively and used the FACAD dataset to train their model.…”

Computer Vision in Human Analysis: From Face and Body to Clothes

“…Finally, two papers focus on the problem of outward appearance and fashion. In particular, Fontanini et al [ 12 ] propose a method for transferring clothing styles across images of people, while Moratelli et al [ 13 ] propose an image captioning approach for fashion retrieval applications.…”

TSIC-CLIP: Traffic Scene Image Captioning Model Based on Clip

“…With the development of Artificial Intelligence, image captioning techniques have been increasingly applied in various fields. Such as medicine [5,28], fashion and e-commerce [19], aided industry [36], and tourism [4]. Moreover, this technology shows immense potential in traffic applications.…”