Efficient Human-Robot Interaction using Deep Learning with Mask R-CNN: Detection, Recognition, Tracking and Segmentation

Le, Than; Huynh, Dang; Pham, Huy

doi:10.1109/icarcv.2018.8581081

Cited by 15 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the field of human-robot interaction, CNNs have been used in many papers [ 43 , 44 , 45 , 46 , 47 , 48 ]. In [ 43 ] a hybrid learning algorithm was proposed to study the reliability of the positioning accuracy of industrial robots more efficiently and accurately.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Others papers such as [ 45 , 46 , 47 ] have used the CNN to recognize facial expressions using social robots. In [ 45 ] a CNN architecture based on emotions for robots was presented.…”

Section: Literature Reviewmentioning

confidence: 99%

“…The authors explained why it may be more effective to use a CNN compared to other methods to have better emotion in robots. In [ 46 ], an integration of a deep neural network (Mask R-CNN) with a mechanical robotic system is proposed. In this way, the system is more robust for human-robot interactive activities.…”

Section: Literature Reviewmentioning

confidence: 99%

See 2 more Smart Citations

Using a Social Robot to Evaluate Facial Expressions in the Wild

Perales

Buades

2020

Sensors

View full text Add to dashboard Cite

In this work an affective computing approach is used to study the human-robot interaction using a social robot to validate facial expressions in the wild. Our global goal is to evaluate that a social robot can be used to interact in a convincing manner with human users to recognize their potential emotions through facial expressions, contextual cues and bio-signals. In particular, this work is focused on analyzing facial expression. A social robot is used to validate a pre-trained convolutional neural network (CNN) which recognizes facial expressions. Facial expression recognition plays an important role in recognizing and understanding human emotion by robots. Robots equipped with expression recognition capabilities can also be a useful tool to get feedback from the users. The designed experiment allows evaluating a trained neural network in facial expressions using a social robot in a real environment. In this paper a comparison between the CNN accuracy and human experts is performed, in addition to analyze the interaction, attention and difficulty to perform a particular expression by 29 non-expert users. In the experiment, the robot leads the users to perform different facial expressions in motivating and entertaining way. At the end of the experiment, the users are quizzed about their experience with the robot. Finally, a set of experts and the CNN classify the expressions. The obtained results allow affirming that the use of social robot is an adequate interaction paradigm for the evaluation on facial expression.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

“…Others papers such as [ 45 , 46 , 47 ] have used the CNN to recognize facial expressions using social robots. In [ 45 ] a CNN architecture based on emotions for robots was presented.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Using a Social Robot to Evaluate Facial Expressions in the Wild

Perales

Buades

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Solving this task plays an important role in scene understanding, evidenced by recent surge of interest in computer vision community [3]- [6]. Among all the object categories considered, human instance has attracted significant attention due to its wide range of real-world applications, such as human-robot interaction [7], human behavior analysis [8], and autonomous driving [9], as well as the unique challenges in human segmentation for general scenes [1].…”

Section: Introductionmentioning

confidence: 99%

PoSeg: Pose-Aware Refinement Network for Human Instance Segmentation

Zhou

2020

IEEE Access

View full text Add to dashboard Cite

Human instance segmentation is a core problem for human-centric scene understanding and segmenting human instances poses a unique challenge to vision systems due to large intra-class variations in both appearance and shape, and complicated occlusion patterns. In this paper, we propose a new poseaware human instance segmentation method. Compared to the previous pose-aware methods which first predict bottom-up poses and then estimate instance segmentation on top of predicted poses, our method integrates both top-down and bottom-up cues for an instance: it adopts detection results as human proposals and jointly estimates human pose and instance segmentation for each proposal. We develop a modular recurrent deep network that utilizes pose estimation to refine instance segmentation in an iterative manner. Our refinement modules exploit pose cues in two levels: as a coarse shape prior and local part attention. We evaluate our approach on two public multi-person benchmarks: OCHuman dataset and COCOPersons dataset. The proposed method surpasses the state-of-the-art methods on OCHuman dataset by 3.0 mAP and on COCOPersons by 6.4 mAP, demonstrating the effectiveness of our approach.

show abstract

“…Deep learning technology is gradually applied in the field of quality control [13,14]. There are also some neural networks that combine different functions to design some special loss functions, which can be classified efficiently while focusing on pixel-level changes [15][16][17][18]. With the increasingly strong recognition ability of a neural network framework, many models can complete multi-task learning [19,20].…”

Section: Introductionmentioning

confidence: 99%

Artificial Auditory Perception Pattern Recognition System Based on Spatiotemporal Convolutional Neural Network

et al. 2019

View full text Add to dashboard Cite

It is difficult to combine human sensory cognition with quality detection to form a pattern recognition system based on human perception. In the future, miniature stepper motor modules will be widely used in advanced intelligent equipment. However, the reducer module based on powder metallurgy parts and the stepper motor may have various defects during operation, with varying definitions of those that affect the user comfort. It is tremendously important to develop an intelligent system to effectively simulate human senses. In this work, an elaborated personification of the perceptual system is proposed to simulate the ventral and flow of the human perception system: two branch systems consisting of a spatiotemporal convolutional neural network (S-CNN) and a concatenated HoppingNet temporal convolutional neural network (T-CNN). To ensure high robustness of the system, we combined principal component analysis (PCA) with the opinions of an experienced quality control (QC) team members to screen the data, and used a bionic ear to simulate human perception characteristics. After repeated comparisons of the tester, the results show that our anthropoid pattern sensing system has high accuracy and robustness for a stepper motor module.

show abstract

Efficient Human-Robot Interaction using Deep Learning with Mask R-CNN: Detection, Recognition, Tracking and Segmentation

Cited by 15 publications

References 16 publications

Using a Social Robot to Evaluate Facial Expressions in the Wild

Using a Social Robot to Evaluate Facial Expressions in the Wild

PoSeg: Pose-Aware Refinement Network for Human Instance Segmentation

Artificial Auditory Perception Pattern Recognition System Based on Spatiotemporal Convolutional Neural Network

Contact Info

Product

Resources

About