Learning to Describe E-Commerce Images from Noisy Online Data

Yashima, Takuya; Okazaki, Naoaki; Inui, Kentaro; Yamaguchi, Kota; Okatani, Takayuki

doi:10.1007/978-3-319-54193-8_6

Cited by 6 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Typically in these approaches, web-crawlers collect easily available noisy multi-modal data [8,12,79] or e-books [17] which is jointly processed for labelling and knowledge extraction. The features are used for diverse applications such as classification and retrieval [68,76] or product description generation [82].…”

Section: Related Workmentioning

confidence: 99%

“…Although trained for classification, these ImageNet-based features have been shown to translate well to other tasks such as segmentation [42], style-transfer [22], NIC. In fact, due to the unavailability of large-scale, task-specific CNN annotations, these ImageNet features have been used for other variants of NIC such as aesthetic captioning [11], stylized captioning [56], product descriptions [82], etc.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Aesthetic Image Captioning From Weakly-Labelled Photographs

Ghosal

Rana

Smolić

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Aesthetic image captioning (AIC) refers to the multimodal task of generating critical textual feedbacks for photographs. While in natural image captioning (NIC), deep models are trained in an end-to-end manner using large curated datasets such as MS-COCO, no such large-scale, clean dataset exists for AIC. Towards this goal, we propose an automatic cleaning strategy to create a benchmarking AIC dataset, by exploiting the images and noisy comments easily available from photography websites. We propose a probabilistic caption-filtering method for cleaning the noisy web-data, and compile a large-scale, clean dataset 'AVA-Captions', ( ∼ 230, 000 images with ∼ 5 captions per image). Additionally, by exploiting the latent associations between aesthetic attributes, we propose a strategy for training a convolutional neural network (CNN) based visual feature extractor, typically the first component of an AIC framework. The strategy is weakly supervised and can be effectively used to learn rich aesthetic representations, without requiring expensive ground-truth annotations. We finally showcase a thorough analysis of the proposed contributions using automatic metrics and subjective evaluations.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Aesthetic Image Captioning From Weakly-Labelled Photographs

Ghosal

Rana

Smolić

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

show abstract

“…(2) the task can be used to infer product categories in the cases when product categorical data is unavailable, noisy, or incomplete [39]; and (3) the design of cross-categorical promotions and product category landing pages [24].…”

Section: Introductionmentioning

confidence: 99%

Extending CLIP for Category-to-image Retrieval in E-commerce

Hendriksen¹,

Bleeker²,

Vakulenko³

et al. 2021

Preprint

View full text Add to dashboard Cite

E-commerce provides rich multimodal data that is barely leveraged in practice. One aspect of this data is a category tree that is being used in search and recommendation. However, in practice, during a user's session there is often a mismatch between a textual and a visual representation of a given category. Motivated by the problem, we introduce the task of category-to-image retrieval in e-commerce and propose a model for the task, CLIP-ITA. The model leverages information from multiple modalities (textual, visual, and attribute modality) to create product representations. We explore how adding information from multiple modalities (textual, visual, and attribute modality) impacts the model's performance. In particular, we observe that CLIP-ITA significantly outperforms a comparable model that leverages only the visual modality and a comparable model that leverages the visual and attribute modality.

show abstract