3-D-Gaze-Based Robotic Grasping Through Mimicking Human Visuomotor Function for People With Motion Impairments

Li, Songpo; Zhang, Xiaoli; Webb, Jeremy D.

doi:10.1109/tbme.2017.2677902

Cited by 55 publications

(34 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human eye tracking data have also been used in the closed loop control of robotic arms. Recently, Li et al (2017) demonstrated how 3D gaze tracking could be used to enable individuals with impaired mobility to control a robotic arm in an intuitive manner. Diverging from traditional gaze tracking approaches that leverage two-dimensional (2D) egocentric camera videos, Li et al presented methods for estimating object location and pose from gaze points reconstructed in 3D.…”

mentioning

confidence: 99%

Exploiting Three-Dimensional Gaze Tracking for Action Recognition During Bimanual Manipulation to Enhance Human–Robot Collaboration

2018

View full text Add to dashboard Cite

Human-robot collaboration could be advanced by facilitating the intuitive, gaze-based control of robots, and enabling robots to recognize human actions, infer human intent, and plan actions that support human goals. Traditionally, gaze tracking approaches to action recognition have relied upon computer vision-based analyses of two-dimensional egocentric camera videos. The objective of this study was to identify useful features that can be extracted from three-dimensional (3D) gaze behavior and used as inputs to machine learning algorithms for human action recognition. We investigated human gaze behavior and gaze-object interactions in 3D during the performance of a bimanual, instrumental activity of daily living: the preparation of a powdered drink. A marker-based motion capture system and binocular eye tracker were used to reconstruct 3D gaze vectors and their intersection with 3D point clouds of objects being manipulated. Statistical analyses of gaze fixation duration and saccade size suggested that some actions (pouring and stirring) may require more visual attention than other actions (reach, pick up, set down, and move). 3D gaze saliency maps, generated with high spatial resolution for six subtasks, appeared to encode action-relevant information. The "gaze object sequence" was used to capture information about the identity of objects in concert with the temporal sequence in which the objects were visually regarded. Dynamic time warping barycentric averaging was used to create a population-based set of characteristic gaze object sequences that accounted for intra-and inter-subject variability. The gaze object sequence was used to demonstrate the feasibility of a simple action recognition algorithm that utilized a dynamic time warping Euclidean distance metric. Averaged over the six subtasks, the action recognition algorithm yielded an accuracy of 96.4%, precision of 89.5%, and recall of 89.2%. This level of performance suggests that the gaze object sequence is a promising feature for action recognition whose impact could be enhanced through the use of sophisticated machine learning classifiers and algorithmic improvements for real-time implementation. Robots capable of robust, realtime recognition of human actions during manipulation tasks could be used to improve quality of life in the home and quality of work in industrial environments.

show abstract

mentioning

confidence: 99%

Exploiting Three-Dimensional Gaze Tracking for Action Recognition During Bimanual Manipulation to Enhance Human–Robot Collaboration

2018

View full text Add to dashboard Cite

show abstract

“…In the early 2000's, the eyetracker was used as a direct substitute for a handheld mouse such that the gaze point on a computer display designates the cursor's position, and blinks function as button clicks (Lin et al, 2006 ; Gajwani and Chhabria, 2010 ). Since 2015, eye gaze has been used to communicate a 3D target position (Li et al, 2015a , 2017 ; Dziemian et al, 2016 ; Li and Zhang, 2017 ; Wang et al, 2018 ; Zeng et al, 2020 ) for directing the movement of the robotic end effector. No action recognition was required, as these methods assumed specific actions in advance, such as reach and grasp (Li et al, 2017 ), write and draw (Dziemian et al, 2016 ), and pick and place (Wang et al, 2018 ).…”

Section: Related Workmentioning

confidence: 99%

Toward Shared Autonomy Control Schemes for Human-Robot Systems: Action Primitive Recognition Using Eye Gaze Features

2020

View full text Add to dashboard Cite

The functional independence of individuals with upper limb impairment could be enhanced by teleoperated robots that can assist with activities of daily living. However, robot control is not always intuitive for the operator. In this work, eye gaze was leveraged as a natural way to infer human intent and advance action recognition for shared autonomy control schemes. We introduced a classifier structure for recognizing low-level action primitives that incorporates novel three-dimensional gaze-related features. We defined an action primitive as a triplet comprised of a verb, target object, and hand object. A recurrent neural network was trained to recognize a verb and target object, and was tested on three different activities. For a representative activity (making a powdered drink), the average recognition accuracy was 77% for the verb and 83% for the target object. Using a non-specific approach to classifying and indexing objects in the workspace, we observed a modest level of generalizability of the action primitive classifier across activities, including those for which the classifier was not trained. The novel input features of gaze object angle and its rate of change were especially useful for accurately recognizing action primitives and reducing the observational latency of the classifier.

show abstract

“…When target points are outside the calibration plane, 2D gaze estimation methods result in an error of gaze points, i.e., the parallel error [20], due to the offset between scene cameras and eyes. To this end, some methods use additional input features related to depth coordinates of gaze points, such as pupil distances [5] or Purkinje images [21]. It is highlighted that the depth error of gaze estimation is generally significant, as these additional features have an indirect and weak correlation with depth coordinates of gaze points.…”

Section: Related Workmentioning

confidence: 99%

“…Gaze tracking is mainly used for attention analysis [1], [2], human-computer interaction [3], and human-robot interaction (HRI) [4], [5]. Li et al [5] recently reported the first attempt to achieve intuitive HRI using only gaze signals, which is helpful for disabled people with upper limb motor impairment, such as amputees and paralyzed patients, to reconstruct their upper limb motion abilities. In mobile applications, such as gaze-based intuitive HRI, headmounted 3D gaze trackers are preferred to table-mounted gaze trackers [6].…”

Section: Introductionmentioning

confidence: 99%

Accurate Regression-Based 3D Gaze Estimation Using Multiple Mapping Surfaces

Wan

Chen

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Accurate 3D gaze estimation using a simple setup remains a challenging issue for headmounted eye tracking. Current regression-based gaze direction estimation methods implicitly assume that all gaze directions intersect at one point called the eyeball pseudo-center. The effect of this implicit assumption on gaze estimation is unknown. In this paper, we find that this assumption is approximate based on a simulation of all intersections of gaze directions, and it is conditional based on a sensitivity analysis of the assumption in gaze estimation. Hence, we propose a gaze direction estimation method with one mapping surface that satisfies conditions of the assumption by configuring one mapping surface and achieving a highquality calibration of the eyeball pseudo-center. This method only adds two additional calibration points outside the mapping surface. Furthermore, replacing the eyeball pseudo-center with an additional calibrated surface, we propose a gaze direction estimation method with two mapping surfaces that further improves the accuracy of gaze estimation. This method improves accuracy on the state-of-the-art method by 20 percent (from a mean error of 1.84 degrees to 1.48 degrees) on a public dataset with a usage range of 1 meter and by 17 percent (from a mean error of 2.22 degrees to 1.85 degrees) on a public dataset with a usage range of 2 meters.

show abstract

3-D-Gaze-Based Robotic Grasping Through Mimicking Human Visuomotor Function for People With Motion Impairments

Cited by 55 publications

References 55 publications

Exploiting Three-Dimensional Gaze Tracking for Action Recognition During Bimanual Manipulation to Enhance Human–Robot Collaboration

Exploiting Three-Dimensional Gaze Tracking for Action Recognition During Bimanual Manipulation to Enhance Human–Robot Collaboration

Toward Shared Autonomy Control Schemes for Human-Robot Systems: Action Primitive Recognition Using Eye Gaze Features

Accurate Regression-Based 3D Gaze Estimation Using Multiple Mapping Surfaces

Contact Info

Product

Resources

About