A Person-Following Shopping Support Robot Based on Human Pose Skeleton Data and LiDAR Sensor

Islam, Md. Matiqul; Lam, Antony; Fukuda, Hisato; Kobayashi, Yoshinori; Kuno, Yoshinori

doi:10.1007/978-3-030-26766-7_2

Cited by 17 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the objectives of the work was to develop a simple, reliable and easy to use system that could provide freedom of movement for elderly and handicapped people. To do so, a person following robot was develop in our previous work [2]. In this work, we provide shopping support facilities for the elderly.…”

Section: Discussionmentioning

confidence: 99%

“…In our previous work [2], we developed a person following shopping support robot. In this paper, we focus on more intelligent shopping support robot that can recognize the customer's shopping behavior and act accordingly.…”

Section: Customer Behavior Model For the Front Of The Shelfmentioning

confidence: 99%

“…The last layer is a Dense layer and it is a fully connected layer. following system are discussed in our previous paper [2]. Our shopping support robot uses a LiDAR sensor about 20 cm high from the floor.…”

Section: Experiments Descriptionmentioning

confidence: 99%

“…Shopping malls or supermarkets typically have many crowded regions. For this reason, in our previous research [2], we developed an autonomous person following robot that can follow a given target person in crowded areas.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An intelligent shopping support robot: understanding shopping behavior from 2D skeleton data using GRU network

Islam

Lam²,

Fukuda

et al. 2019

Robomech J

Self Cite

View full text Add to dashboard Cite

In supermarkets or grocery, a shopping cart is a necessary tool for shopping. In this paper, we have developed an intelligent shopping support robot that can carry a shopping cart while following its owners and provide the shopping support by observing the customer's head orientation, body orientation and recognizing different shopping behaviors. Recognizing shopping behavior or the intensity of such action is important for understanding the best way to support the customer without disturbing him or her. This system also liberates elderly and disabled people from the burden of pushing shopping carts, because our proposed shopping cart is essentially a type of autonomous mobile robots that recognizes its owner and following him or her. The proposed system discretizes the head and body orientation of customer into 8 directions to estimate whether the customer is looking or turning towards a merchandise shelf. From the robot's video stream, a DNN-based human pose estimator called OpenPose is used to extract the skeleton of 18 joints for each detected body. Using this extracted body joints information, we built a dataset and developed a novel Gated Recurrent Neural Network (GRU) topology to classify different actions that are typically performed in front of shelves: reach to shelf, retract from shelf, hand in shelf, inspect product, inspect shelf. Our GRU network model takes series of 32 frames skeleton data then gives the prediction. Using cross-validation tests, our model achieves an overall accuracy of 82%, which is a significant result. Finally, from the customer's head orientation, body orientation and shopping behavior recognition we develop a complete system for our shopping support robot.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Customer Behavior Model For the Front Of The Shelfmentioning

confidence: 99%

Section: Experiments Descriptionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An intelligent shopping support robot: understanding shopping behavior from 2D skeleton data using GRU network

Islam

Lam²,

Fukuda

et al. 2019

Robomech J

Self Cite

View full text Add to dashboard Cite

show abstract

“…A new automated smart cart system for modern shopping centres (H. Bello-Salau) 2029 as well as some android application-controlled approaches were presented in [10]- [14]. While, other efforts addressed automated billing system issues and developed predictive algorithms to suggests items to customers [15]- [17].…”

Section: Introductionmentioning

confidence: 99%

A new automated smart cart system for modern shopping centres

et al. 2021

View full text Add to dashboard Cite

Modern shopping centres are undoubtedly a beehive of intense shopping activities. However, customers are often plagued by salient challenges, which may include fatigue derived from pushing trolleys around the mall and prolonged sorting of bills by the cashier. These shopping challenges could be daunting for the elderly, disabled, pregnant and nursing mothers. In this paper, we addressed these shopping challenges by developing an autonomous shopping cart with the following characteristics; (1) it follows the customer’s movement relieving the need to push a cart, (2) it bills automatically all stock placed in the cart, (3) it prompts the customer to make payment and updates each stock via a local database. Our design adopts a Raspberry Pi, a camera and a few direct current motors programmed to achieve autonomy. We used an open-source cross platform software called XAMPP to create the database and used RFID tags to bill the items placed in the cart automatically. The system updates payments and communicates these transactions to a local database via nRF24 wireless transceivers. The experimental tests conducted demonstrate that our system successfully followed customers accurately within the mall. We consider our design a major contribution to the vision of automated shopping systems for the near future.

show abstract

Person-Following Shopping Support Robot Using Kinect Depth Camera Based on 3D Skeleton Tracking

Islam

Lam²,

Fukuda

et al. 2020

Intelligent Computing Methodologies

View full text Add to dashboard Cite

A Person-Following Shopping Support Robot Based on Human Pose Skeleton Data and LiDAR Sensor

Cited by 17 publications

References 8 publications

An intelligent shopping support robot: understanding shopping behavior from 2D skeleton data using GRU network

An intelligent shopping support robot: understanding shopping behavior from 2D skeleton data using GRU network

A new automated smart cart system for modern shopping centres

Person-Following Shopping Support Robot Using Kinect Depth Camera Based on 3D Skeleton Tracking

Contact Info

Product

Resources

About