Deep understanding of shopper behaviours and interactions using RGB-D vision

Abstract

In retail environments, understanding how shoppers move about in a store’s spaces and interact with products is very valuable. While the retail environment has several favourable characteristics that support computer vision, such as reasonable lighting, the large number and diversity of products sold, as well as the potential ambiguity of shoppers’ movements, mean that accurately measuring shopper behaviour is still challenging. Over the past years, machine-learning and feature-based tools for people counting as well as interactions analytic and re-identification were developed with the aim of learning shopper skills based on occlusion-free RGB-D cameras in a top-view configuration. However, after moving into the era of multimedia big data, machine-learning approaches evolved into deep learning approaches, which are a more powerful and efficient way of dealing with the complexities of human behaviour. In this paper, a novel VRAI deep learning application that uses three convolutional neural networks to count the number of people passing or stopping in the camera area, perform top-view re-identification and measure shopper–shelf interactions from a single RGB-D video flow with near real-time performances has been introduced. The framework is evaluated on the following three new datasets that are publicly available: TVHeads for people counting, HaDa for shopper–shelf interactions and TVPR2 for people re-identification. The experimental results show that the proposed methods significantly outperform all competitive state-of-the-art methods (accuracy of 99.5% on people counting, 92.6% on interaction classification and 74.5% on re-id), bringing to different and significative insights for implicit and extensive shopper behaviour analysis for marketing applications.

Representation Learning for Fine-Grained Change Detection

“…FG analysis is a long-standing and fundamental problem because small inter-class variations in the phenomenon of interest can often be masked by large intra-class variations de to ancillary data [ 2 ]. However, it is an important problem and has become ubiquitous in diverse CD applications such as automatic biodiversity monitoring [ 3 ], climate change evaluation [ 4 ], intelligent retail [ 5 ], intelligent transportation [ 6 ], and many more.…”

Human-Robot Interaction and Collaboration (HRI-C) Utilizing Top-View RGB-D Camera System

“…While this configuration of specific measurement was utilized to track human beings [6], to our knowledge and based on the conducted literature review, it was not previously studied in the context of human-robot interaction and collaboration. Some researcher [7][8][9][10][11][12][13][14][15] analyzed the literature review and found that most of the RGB-D use was meant for human identification and tracking, human activity recognition, human behavior analysis for shopping and security purposes, intelligent health care systems, detecting defects in produce and animal recognition, also a data-based had been developed to summarize all these uses and algorithms. It was proven that the top-view RGB-D cameras can be utilized successfully in several applications where behaviors and interactions can be analyzed and they are very attractive due to their affordability and the sufficient information extracted from the provided pictures or live feed.…”

A Hierarchy-Based System for Recognizing Customer Activity in Retail Environments

“…However, in traditional retail environments, available information tends to be limited to purchase records of customers, which cannot reveal any details of CA such as movement of customers in store spaces and interaction of customers with products [ 3 ]. Therefore, several systems have been proposed for customer activity recognition (CAR) from videos taken by in-store cameras, and their efforts are devoted to customer detection, counting, re-identification, tracking, behavior recognition, and so on [ 4 ].…”

“…Most of the existing CAR systems use machine learning based end-to-end (E2E) models due to their remarkable accuracy in CAR [ 4 ]. Usually such E2E models in CAR systems are constructed as complete CAR models, each of which is trained to recognize particular types of CA from input videos of specific store spaces.…”