Evaluation of Vision-Based Hand Tool Tracking Methods for Quality Assessment and Training in Human-Centered Industry 4.0

Abstract

Smart industrial workstations for the training and evaluation of workers are an innovative approach to face the problems of manufacturing quality assessment and fast training. However, such products do not implement algorithms that are able to accurately track the pose of a hand tool that might also be partially occluded by the operator’s hands. In the best case, the already proposed systems roughly track the position of the operator’s hand center assuming that a certain task has been performed if the hand center position is close enough to a specified area. The problem of the pose estimation of 3D objects, including the hand tool, is an open and debated problem. The methods that lead to high accuracies are time consuming and require a 3D model of the object to detect, which is why they cannot be adopted for a real-time training system. The rise in deep learning has stimulated the search for better-performing vision-based solutions. Nevertheless, the problem of hand tool pose estimation for assembly and training procedures appears to not have been extensively investigated. In this study, four different vision-based methods based, respectively, on ArUco markers, OpenPose, Azure Kinect Body Tracking and the YOLO network have been proposed in order to estimate the position of a specific point of interest of the tool that has to be tracked in real-time during an assembly or maintenance procedure. The proposed approaches have been tested on a real scenario with four users handling a power drill simulating three different conditions during an assembly procedure. The performance of the methods has been evaluated and compared with the HTC Vive tracking system as a benchmark. Then, the advantages and drawbacks in terms of the accuracy and invasiveness of the method have been discussed. The authors can state that OpenPose is the most robust proposal arising from the study. The authors will investigate the OpenPose performance in more depth in further studies. The framework appears to be very interesting regarding its integration into a smart workstation for quality assessment and training.

On the Evaluation of Diverse Vision Systems towards Detecting Human Pose in Collaborative Robot Applications

“…Related approaches that identify the orientation of human body poses may be used in cases of Human–Robot Collaboration for real-time decision making and path planning to carry out tasks. Similarly, De Feudis et al [ 43 ] assessed four different vision systems for hand tool pose estimation: ArUco, OpenPose, Azure Kinect Body Tracking, and YOLO network were used with HTC Vive as a benchmarking system. Further, in a study presented in [ 44 ], Azure Kinect and Intel RealSense D435i were compared where the Intel RealSense was reported to show poorer performance in the estimation after 2 m, while the Azure Kinect performed better.…”

“…The experimentation involved three different motion scenarios of a human operator handling a cordless drill with its mandrel considered as the point of interest to be tracked [ 43 ]. The mean square point-to-point distance (D.RMS) and the multivariate R 2 were used as the accuracy evaluation criteria.…”

Impact of Augmented Reality on Assistance and Training in Industry 4.0: Qualitative Evaluation and Meta-Analysis

“…In terms of quality control analysis, AR has emerged as a valuable tool, allowing for more accurate and detailed assessment. A study by De Feudis et al [78] explored the application of AR in quality assessment, making use of hand tool tracking through machine vision techniques. AR, with its capacity to provide real-time visualisations, is distinguished by its adaptability and flexibility, enabling the constant optimisation of quality protocols and the agile adaptation of quality standards to the changing dynamics of the industrial sector.…”

Combination of Deep Cross-Stage Partial Network and Spatial Pyramid Pooling for Automatic Hand Detection

“…The Body Tracking SDK is built on a complicated deep learning model that allows body segmentation, human skeleton reconstruction, human body instance detection, and real-time body tracking. A power drill chunk pose can be best estimated by looking at the n.8 (left hand) and n.15 (right hand) kinect hand joints, according to [21]. As a result of multiple testing, these joints were ruled out due of their poor orientation predictions.…”