End-to-End Human-Gaze-Target Detection with Transformers

Abstract

This paper proposes an efficient and effective method for joint gaze location detection (GL-D) and gaze object detection (GO-D), i.e., gaze following detection. Current approaches frame GL-D and GO-D as two separate tasks, employing a multi-stage framework where human head crops must first be detected and then be fed into a subsequent GL-D sub-network, which is further followed by an additional object detector for GO-D. In contrast, we reframe the gaze following detection task as detecting human head locations and their gaze followings simultaneously, aiming at jointly detect human gaze location and gaze object in a unified and single-stage pipeline. To this end, we propose GTR, short for Gaze following detection TRansformer, streamlining the gaze following detection pipeline by eliminating all additional components, leading to the first unified paradigm that unites GL-D and GO-D in a fully end-to-end manner. GTR enables an iterative interaction between holistic semantics and human head features through a hierarchical structure, inferring the relations of salient objects and human gaze from the global image context and resulting in an impressive accuracy. Concretely, GTR achieves a 12.1 mAP gain (25.1%) on GazeFollowing and a 18.2 mAP gain (43.3%) on VideoAttentionTarget for GL-D, as well as a 19 mAP improvement (45.2%) on GOO-Real for GO-D. Meanwhile, unlike existing systems detecting gaze following sequentially due to the need for a human head as input, GTR has the flexibility to comprehend any number of people's gaze followings simultaneously, resulting in high efficiency. Specifically, GTR introduces over a ×9 improvement in FPS and the relative gap becomes more pronounced as the human number grows.

Gaze Target Detection by Merging Human Attention and Activity Cues

“…Gaze Target Detection Methods Furthermore, we conduct comparisons with five recent methods: Chen (Chen et al 2021), Fang (Fang et al 2021), Tu (Tu et al 2022), Bao (Bao, Liu, and Yu 2022), and Miao (Miao, Hoai, and Samaras 2023). These methods have all demonstrated notable performance within the confines of within-dataset evaluations.…”

“…For our method, we use the pre-trained lightweight body pose estimator RTMPose (Jiang et al 2023) and object detector YOLOv3 (Redmon et al 2016). On the other hand, competing methods introduced some other modules, e.g., face detection and depth estimation from the scene (Fang et al 2021), body pose estimation and 3D reconstruction from the scene (Bao, Liu, and Yu 2022), ViT backbone (Tu et al 2022). In order to measure their computation complexity, we also select recent high-speed implementations for them, and compared their inference speed on a single NVIDIA Titan XP GPU.…”

Human–object interaction prediction in videos through gaze following

“…The main reason could be that the spatio-temporal transformer is trained with noisy gaze cues as the VidHOI dataset lacks ground-truth gaze annotations. The performance of the adopted gaze following model (Chong et al, 2020) might be a limitation of our framework, but could be improved by leveraging more recent works in that field, such as (Tu et al, 2022a;Fang et al, 2021). In addition, even though the gaze does not result in big improvement, other extensions we proposed in the spatio-temporal transformer still boost the model performance and allow us to achieve state-of-the-art in HOI detection and anticipation in videos.…”

Analyzing students' attention by gaze tracking and object detection in classroom teaching

“…Zhong et al (2021) proposed a one-stage method, namely glance and gaze network, which adaptively simulated a set of action-aware points through glance and gaze steps. Tu et al (2022) presented an effective and efficient method for human-gaze-target detection and gaze following based on determination of the relations of salient objects and the human gaze from the global image context.…”