Prediction of user's grasping intentions based on eye-hand coordination

Abstract

Eye-hand coordination is a primordial reach-tograsp action performed by a human hand when reaching for an object. This paper proposes the use of a visual sensor which allows the simultaneous analysis of hand and eye motions in order to recognize the reach-to-grasp movement, i.e. to predict the grasping gesture. This solution fuses two viewpoints taken from the user's perspective. First, by using an eye-tracker device attached to the user's head; and second, by utilizing a wearable camera attached to the user's hand. The information from these two viewpoints is used to characterize multiple hand movements in conjunction with eye-gaze movements through a HiddenMarkov Model framework. In various experiments, we show that combining these two sources of information allows the prediction of a reach-to-grasp movement as well as the desired object.

AI and ML for Human-Robot Cooperation in Intelligent and Flexible Manufacturing

“…The aim of this process is to continuously monitoring the operator's actions on real-time. The action identification can be driven by different cues, like gestures (Carrasco and Clady 2010), scene objects being manipulated (Koppula and Saxena 2015) or environmental information (Casalino et al 2018). Action recognition has been also extensively studied in terms of whole body motion tracking and segmentation (Natola et al 2015;Tome et al 2017).…”

Operator Awareness in Human–Robot Collaboration Through Wearable Vibrotactile Feedback

“…We think that the same problem can be mitigated by considering also other observations rather than only the gaze, as for example one hand trajectory. This approach was followed in [26], where a hidden Markov model (HMM) was adopted. The HMM allows to make inference about a sliding temporal window of observations.…”