Automated Recognition of Hand Gestures for Crane Rigging using Data Gloves in Virtual Reality

Abstract

Construction safety training through Virtual Reality (VR) environments offers workers an interactive and immersive experience. Many complex interactions and realistic scenarios are possible in VR using accessories such as data gloves and trackers which allow data recording on incidents. For example, misinterpretation of hand signals construction workers give during operations may result in severe accidents on construction sites. This study proposes automatic gesture recognition to identify hand signals for crane rigging operations in VR training. The developed gesture recognition algorithm tracks information from a data glove and a tracker. Preliminary data on movements and orientation of hands and fingers were recorded. Mathematical models of hand gestures were created based on finger movement data. Gesture rules were created based on the rotation and orientation of the hand. The gesture models were combined with the gesture rules to develop the algorithm for automated gesture recognition. Final experiments estimated the efficacy of the proposed method in automatically recognizing crane rigging signals in real-time. The performance is evaluated by comparing the identified hand gestures with independently created ground truth labels. The proposed method identified static hand gestures with an average accuracy of 96.55 percent. This method recognizes the gestures along with the hand movements and displays the results in real-time equivalent to dynamic gesture recognition. More refined dynamic gesture recognition based on this method is in progress.