We Know Where They Are Looking at From the RGB-D Camera: Gaze Following in 3D

Hu, Zhengxi; Yang, Dingye; Cheng, Shilei; Zhou, Lei; Wu, Shichao; Liu, Jingtai

doi:10.1109/tim.2022.3160534

Cited by 12 publications

(6 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Solution for conflicted demonstrations: To resolve the conflicted labels from different experts, a function for conflict resolution is executed after each rollout j before the incorporation of D j into D. The conflict resolution takes D j , D, and σ t as inputs. We use cosine similarity to identify and select similar observations due to its wide applications in similarity detection for various sensors that are commonly seen in robotics, including LiDAR scans [33,34] and RGBD camera [35,36]. To efficiently leverage parallel processing, the cosine similarities Θ between all observations O j in D j and all observations O in D is calculated by the dot product of O and O j divided by element-wise multiplication of the Euclidean norms of O and O j :…”

Section: Methodsmentioning

confidence: 99%

MEGA-DAgger: Imitation Learning with Multiple Imperfect Experts

Sun¹,

Yang²,

Mangharam³

2023

Preprint

View full text Add to dashboard Cite

Imitation learning has been widely applied to various autonomous systems thanks to recent development in interactive algorithms that address covariate shift and compounding errors induced by traditional approaches like behavior cloning. However, existing interactive imitation learning methods assume access to one perfect expert. Whereas in reality, it is more likely to have multiple imperfect experts instead. In this paper, we propose MEGA-DAgger, a new DAgger variant that is suitable for interactive learning with multiple imperfect experts. First, unsafe demonstrations are filtered while aggregating the training data, so the imperfect demonstrations have little influence when training the novice policy. Next, experts are evaluated and compared on scenarios-specific metrics to resolve the conflicted labels among experts. Through experiments in autonomous racing scenarios, we demonstrate that policy learned using MEGA-DAgger can outperform both experts and policies learned using the state-of-the-art interactive imitation learning algorithm. The supplementary video can be found at https://youtu.be/pYQiPSHk6dU.

show abstract

Section: Methodsmentioning

confidence: 99%

MEGA-DAgger: Imitation Learning with Multiple Imperfect Experts

Sun¹,

Yang²,

Mangharam³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Although Deep Neural Networks (DNNs) have been implemented in visual trackers such as Caelles et al ( 2017 ), Wang et al ( 2019 ), Lukezic et al ( 2020 ), and Koide et al ( 2020 ) that produce high frames per second (FPS) ROI trackers, these systems may not adequately track the target during substantial pose variations. There have been efforts to create a more robust representation of a person's position in 3D space using plane and height estimation techniques (Chou and Nakajima, 2017 ; Jiang et al, 2018 ; Zou and Lan, 2019 ; Hu et al, 2022 ). Still, they predominantly yield an approximation of the person's position.…”

Section: Related Workmentioning

confidence: 99%

“…To augment the precision and responsiveness of these robotic applications, the present research introduces a novel end-to-end Deep Neural Network (DNN) for person-following. This approach enables pixel-level tracking of the target individual and integrates a real-time future motion estimation function, facilitating the robot's capacity to anticipate and swiftly react to the individual's movements (Lin et al, 2012 ; Cosgun et al, 2013 ; Cheng et al, 2019 ; Koide et al, 2020 ; Hu et al, 2022 ).…”

Section: Introductionmentioning

confidence: 99%

Tracking by segmentation with future motion estimation applied to person-following robots

Jiang,

Cui,

Wei

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

Person-following is a crucial capability for service robots, and the employment of vision technology is a leading trend in building environmental understanding. While most existing methodologies rely on a tracking-by-detection strategy, which necessitates extensive datasets for training and yet remains susceptible to environmental noise, we propose a novel approach: real-time tracking-by-segmentation with a future motion estimation framework. This framework facilitates pixel-level tracking of a target individual and predicts their future motion. Our strategy leverages a single-shot segmentation tracking neural network for precise foreground segmentation to track the target, overcoming the limitations of using a rectangular region of interest (ROI). Here we clarify that, while the ROI provides a broad context, the segmentation within this bounding box offers a detailed and more accurate position of the human subject. To further improve our approach, a classification-lock pre-trained layer is utilized to form a constraint that curbs feature outliers originating from the person being tracked. A discriminative correlation filter estimates the potential target region in the scene to prevent foreground misrecognition, while a motion estimation neural network anticipates the target's future motion for use in the control module. We validated our proposed methodology using the VOT, LaSot, YouTube-VOS, and Davis tracking datasets, demonstrating its effectiveness. Notably, our framework supports long-term person-following tasks in indoor environments, showing promise for practical implementation in service robots.

show abstract

“…Fang et al [10] used depth to potentially disambiguate attention targets by inferring whether a person is looking toward their foreground or background, obtaining the best results reported so far on the GazeFollow and VideoAtten-tionTarget datasets. Recently, Hu et al [15] used depth information to perform gaze target prediction in 3D. As far as we are aware, ours is the first work to use both pose and depth information, and to study the privacy preserving situation.…”

Section: Gaze Target Predictionmentioning

confidence: 99%

A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings

Gupta

Samy

Odobez

2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Predicting where a person is looking is a complex task, requiring to understand not only the person's gaze and scene content, but also the 3D scene structure and the person's situation (are they manipulating? interacting or observing others? attentive?) to detect obstructions in the line of sight or apply attention priors that humans typically have when observing others. In this paper, we hypothesize that identifying and leveraging such priors can be better achieved through the exploitation of explicitly derived multimodal cues such as depth and pose. We thus propose a modular multimodal architecture allowing to combine these cues using an attention mechanism. The architecture can naturally be exploited in privacy-sensitive situations such as surveillance and health, where personally identifiable information cannot be released. We perform extensive experiments on the GazeFollow and VideoAtten-tionTarget public datasets, obtaining state-of-the-art performance and demonstrating very competitive results in the privacy setting case. 1

show abstract

We Know Where They Are Looking at From the RGB-D Camera: Gaze Following in 3D

Cited by 12 publications

References 55 publications

MEGA-DAgger: Imitation Learning with Multiple Imperfect Experts

MEGA-DAgger: Imitation Learning with Multiple Imperfect Experts

Tracking by segmentation with future motion estimation applied to person-following robots

A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings

Contact Info

Product

Resources

About