Prediction-Based Human-Robot Collaboration in Assembly Tasks Using a Learning from Demonstration Model

Abstract

Most robots are programmed to carry out specific tasks routinely with minor variations. However, more and more applications from SMEs require robots work alongside their counterpart human workers. To smooth the collaboration task flow and improve the collaboration efficiency, a better way is to formulate the robot to surmise what kind of assistance a human coworker needs and naturally take the right action at the right time. This paper proposes a prediction-based human-robot collaboration model for assembly scenarios. An embedded learning from demonstration technique enables the robot to understand various task descriptions and customized working preferences. A state-enhanced convolutional long short-term memory (ConvLSTM)-based framework is formulated for extracting the high-level spatiotemporal features from the shared workspace and predicting the future actions to facilitate the fluent task transition. This model allows the robot to adapt itself to predicted human actions and enables proactive assistance during collaboration. We applied our model to the seats assembly experiment for a scale model vehicle and it can obtain a human worker’s intentions, predict a coworker’s future actions, and provide assembly parts correspondingly. It has been verified that the proposed framework yields higher smoothness and shorter idle times, and meets more working styles, compared to the state-of-the-art methods without prediction awareness.

Integrating intention-based systems in human-robot interaction: a scoping review of sensors, algorithms, and trust

“…The ability to identify hand movement intention is prominent for robots on collaborated assembly line in HRI scenario. Zhang et al (2022) proposed a method to predict human hand motion during an assembly task to improve collaboration flow and efficiency. The design utilizes an RGB camera mounted over the robot, facing downwards at the working area on the table.…”

“…When analyzing the data extracted from the literature and presented in Table 2 , it becomes clear that the most popular sensor used in designs is the RGB camera ( Fang et al, 2017 ; Li et al, 2018 ; Gardner et al, 2020 ; Jaouedi et al, 2020 ; Mohammadi Amin et al, 2020 ; Chiu et al, 2021 ; Ding and Zheng, 2022 ; Poulose et al, 2022 ; Tsitos et al, 2022 ; Zhang et al, 2022 ). This is likely due to the widespread use of image recognition applications in recent years, as well as the relative affordability and accessibility of RGB cameras in work environments.…”

Skeleton-based Human Activity Classification in Sparse Image Sequences

“…Human activity recognition has recently caught the attention of the computer vision community since it drives real-world applications that make our life better and safer, such as human-computer interaction in robotics and gaming, video surveillance, and social activity recognition [1]. For example, new robotic applications try to predict human activity patterns in order to let the robot early infer when a speci ic collaborative operation will be requested by the human [2,3]. In video surveillance, human activity classi ication can be integrated with probabilistic prediction models, in order to infer the ongoing activity [4].…”

Hole-Finding Learning Strategy for a Robot Assembly With Keyed Circular Peg

“…Kim et al [12] trained a position-force mapping model for a 3-mm square area using a clustering algorithm and achieved a model error of less than 0.35 mm. A peg-hole data model uses machine learning methods to train contact state data offline and predict the current state online [13][14][15]. This model is updated in real time for online learning through data input from the current contact state [16][17][18][19][20].…”