Real-time multitask multihuman–robot interaction based on context awareness

Abstract

This study presents a novel context awareness multihuman–robot interaction (MHRI) system that allows multiple operators to interact with a robot. In the system, a monocular multihuman 3D pose estimator is first developed with the convolutional neural network. The estimator first regresses a set of 2D joints representations of body parts and then restores the 3D joints positions based on these 2D representations. Further, the 3D joints are assigned to the corresponding individual with a priority–redundancy association algorithm. The whole 3D pose of each person is reconstructed in real time, even in crowded scenes containing both self-occlusion of the body and inter-person occlusion. Then, the identities of multiple persons are recognized with action context and 3D skeleton tracking to improve interactive efficiency. For context-awareness multitask interaction, the robot control strategy is designed based on target goal generation and correction. The generated goal is taken as a reference to the model predictive controller (MPC) to generate motion trajectory. Different interactive requirements are adapted by adjusting the weight parameters of the energy function of the MPC controller. Multihuman–robot interactive experiments, including dynamic obstacle avoidance (human–robot safety) and cooperative handling, demonstrate the feasibility and effectiveness of the MHRI, and the safety and collaborative efficiency of the system are evaluated with HRI metrics.

Data augmentation by separating identity and emotion representations for emotional gait recognition

“…They believed gait-based identity and emotion recognition are interrelated tasks that are favorable for jointly learning. The MTL models entangle information between the tasks to capture the joint dependencies from the multi-labels of the training data [20]. However, there has also been a noticeable absence of studies on MTL for emotional gait, mainly due to the lack of gait datasets annotated with both emotion and identity labels.…”