Detecting Attended Visual Targets in Video

Chong, Eunji; Wang, Yongxin; Ruiz, Nataniel; Rehg, James M.

doi:10.1109/cvpr42600.2020.00544

Cited by 105 publications

(98 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides gaze estimation, in Sec. A of the supplementary, we also show the utility of adding LAEO labels to the task of in-the-wild visual target attention prediction [4] in a semi-supervised setting.…”

Section: Methodsmentioning

confidence: 87%

“…Results Following Chong et al [4], we evaluate the area under the curve (AUC) for correct target location prediction (within a pre-specified distance threshold on the image 3. Improvements to the VATnet baseline [4] by adding weak supervision from the AVA-LAEO dataset using the best configuration of LAEO loss functions described in Table 1 of the main paper.…”

Section: Discussionmentioning

confidence: 99%

“…plane), the L2 distance between the predicted and ground truth target locations in the scene and the out-of-frame prediction's average precision (AP). We report the scores on the VAT test dataset, averaged across training epochs 2-30, both for the author's original method [4] and our proposed modification. Table 3 shows the benefits of jointly training with the AVA-LAEO and VAT datasets.…”

Section: Discussionmentioning

confidence: 99%

“…Chong et al [3] improve performance on the static gaze following task further by jointly training to predict 3D gaze direction using the EYEDIAP dataset [9], and by explicitly predicting whether the target is in frame. This work is also extended to video data in [4]. Much like the task of physically unconstrained gaze estimation, gaze following also involves viewing human subjects in all head poses, in diverse environments, and from larger distances.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Weakly-Supervised Physically Unconstrained Gaze Estimation

Kothari¹,

Mello²,

Iqbal³

et al. 2021

Preprint

View full text Add to dashboard Cite

A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios. In contrast, videos of human interactions in unconstrained environments are abundantly available and can be much more easily annotated with frame-level activity labels. In this work, we tackle the previously unexplored problem of weaklysupervised gaze estimation from videos of human interactions. We leverage the insight that strong gaze-related geometric constraints exist when people perform the activity of "looking at each other" (LAEO). To acquire viable 3D gaze supervision from LAEO labels, we propose a training algorithm along with several novel loss functions especially designed for the task. With weak supervision from two large scale CMU-Panoptic and AVA-LAEO activity datasets, we show significant improvements in (a) the accuracy of semisupervised gaze estimation and (b) cross-domain generalization on the state-of-the-art physically unconstrained in-the-wild Gaze360 gaze estimation benchmark. We open source our code at https://github.com/NVlabs/weaklysupervised-gaze.

show abstract

Section: Methodsmentioning

confidence: 87%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Weakly-Supervised Physically Unconstrained Gaze Estimation

Kothari¹,

Mello²,

Iqbal³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent advances in eye-tracking technologies offer the capability to measure human visual attention [3,9]. Based on physiology and psychology research [5], human attention serves as an information filter and prioritization strategy [5,28], which selectively determines the allocation of cognitive resources.…”

Section: Introductionmentioning

confidence: 99%

MemX

Chang

Zhao

Dong

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

This work presents MemX: a biologically-inspired attention-aware eyewear system developed with the goal of pursuing the long-awaited vision of a personalized visual Memex. MemX captures human visual attention on the fly, analyzes the salient visual content, and records moments of personal interest in the form of compact video snippets. Accurate attentive scene detection and analysis on resource-constrained platforms is challenging because these tasks are computation and energy intensive. We propose a new temporal visual attention network that unifies human visual attention tracking and salient visual content analysis. Attention tracking focuses computation-intensive video analysis on salient regions, while video analysis makes human attention detection and tracking more accurate. Using the YouTube-VIS dataset and 30 participants, we experimentally show that MemX significantly improves the attention tracking accuracy over the eye-tracking-alone method, while maintaining high system energy efficiency. We have also conducted 11 in-field pilot studies across a range of daily usage scenarios, which demonstrate the feasibility and potential benefits of MemX.

show abstract

PerimetryNet: A multiscale fine grained deep network for three‐dimensional eye gaze estimation using visual field analysis

Wang

Zhou

et al. 2023

Computer Animation & Virtual

View full text Add to dashboard Cite

Three‐dimensional gaze estimation aims to reveal where a person is looking, which plays an important role in identifying users' point‐of‐interest in terms of the direction, attention and interactions. Appearance‐based gaze estimation methods could provide relatively unconstrained gaze tracking from commodity hardware. Inspired by medical perimetry test, we have proposed a multiscale framework with visual field analysis branch to improve estimation accuracy. The model is based on the feature pyramids and predicts vision field to help gaze estimation. In particular, we analysis the effect of the multiscale component and the visual field branch on challenging benchmark datasets: MPIIGaze and EYEDIAP. Based on these studies, our proposed PerimetryNet significantly outperforms state‐of‐the‐art methods. In addition, the multiscale mechanism and visual field branch can be easily applied to existing network architecture for gaze estimation. Related code would be available at public repository https://github.com/gazeEs/PerimetryNet.

show abstract

Detecting Attended Visual Targets in Video

Cited by 105 publications

References 44 publications

Weakly-Supervised Physically Unconstrained Gaze Estimation

Weakly-Supervised Physically Unconstrained Gaze Estimation

MemX

PerimetryNet: A multiscale fine grained deep network for three‐dimensional eye gaze estimation using visual field analysis

Contact Info

Product

Resources

About