NLP Cross-Domain Recognition of Retail Products

Pettersson, Tobias; Oucheikh, Rachid; Löfström, Tuwe

doi:10.1145/3529399.3529436

“…In the other fusion models, an image-based model and a multimodal model have to be trained separately to achieve the same purpose. In [9], it is shown that text-based grocery product recognition models are robust to domain adaptation, while the performance of image-based models degrades significantly. This domain adaptation aspect is a practical problem when a system is installed in a new grocery store or if the camera type or camera position changes in an existing one.…”

Section: Discussion and Recommendationsmentioning

confidence: 99%

“…For the recognition of the product package text in grocery products, BERT [85] with a small classification head [9] has shown notably better classification results compared to other methods using GloVe embeddings [86]. Therefore, we use three types of text models based on the Transformer architecture used in BERT; the baseline model BERT [85], the optimized DistilBERT [87] model, and the more accurate DeBERTa [88] model.…”

Section: Model Selectionmentioning

confidence: 99%

“…This approach has been proposed in two recent surveys (see [6,7]) and listed as an important research direction to improve fine-grained recognition of grocery products [6,7]. In fact, recent work has shown that using extracted textual data from packages as input to natural language processing (NLP) models results in a robust text classifier with high accuracy, see [8,9]. Furthermore, Wei et al [7] state that no specific datasets exist for fine-grained product recognition.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Fine-Grained Grocery Product Recognition Using Image and Ocr Text

Pettersson

¹

,

Riveiro

²

,

Löfström

³

2023

Preprint

Self Cite

1

0

View full text Add to dashboard Cite

Multimodal fine-grained grocery product recognition using image and OCR text

Pettersson,

Riveiro,

Löfström

2024

Machine Vision and Applications

0

View full text Add to dashboard Cite

Automatic recognition of grocery products can be used to improve customer flow at checkouts and reduce labor costs and store losses. Product recognition is, however, a challenging task for machine learning-based solutions due to the large number of products and their variations in appearance. In this work, we tackle the challenge of fine-grained product recognition by first extracting a large dataset from a grocery store containing products that are only differentiable by subtle details. Then, we propose a multimodal product recognition approach that uses product images with extracted OCR text from packages to improve fine-grained recognition of grocery products. We evaluate several image and text models separately and then combine them using different multimodal models of varying complexities. The results show that image and textual information complement each other in multimodal models and enable a classifier with greater recognition performance than unimodal models, especially when the number of training samples is limited. Therefore, this approach is suitable for many different scenarios in which product recognition is used to further improve recognition performance. The dataset can be found at https://github.com/Tubbias/finegrainocr.

show abstract

Multimodal Fine-Grained Grocery Product Recognition using Image and OCR Text

Pettersson,

Riveiro,

Löfström

2023

Preprint

Self Cite

0

View full text Add to dashboard Cite

Automatic recognition of grocery products can be used to improve customer flow at checkouts and reduce labor costs and store losses. Product recognition is, however, a challenging task for machine learning-based solutions due to the large number of products and their variations in appearance. In this work, we tackle the challenge of fine-grained product recognition by first extracting a large dataset from a grocery store containing products that are only differentiable by subtle details. Then, we propose a multimodal product recognition approach that uses product images with extracted OCR text from packages to improve fine-grained recognition of grocery products. We evaluate several image and text models separately and then combine them using different multimodal models of varying complexities. The results show that image and textual information complement each other in multimodal models and enable a classifier with greater recognition performance than unimodal models, especially when the number of training samples is limited. Therefore, this approach is suitable for many different scenarios in which product recognition is used to further improve recognition performance.

show abstract

NLP Cross-Domain Recognition of Retail Products

Cited by 4 publications

References 10 publications

Multimodal Fine-Grained Grocery Product Recognition Using Image and Ocr Text

Multimodal Fine-Grained Grocery Product Recognition Using Image and Ocr Text

Multimodal fine-grained grocery product recognition using image and OCR text

Multimodal Fine-Grained Grocery Product Recognition using Image and OCR Text

Contact Info

Product

Resources

About