Visual attention prediction improves performance of autonomous drone racing agents

Abstract

Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural networks’ performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Comparing success rates for completing a challenging race track by autonomous flight, our results show that the attention-prediction based controller (88% success rate) outperforms the RGB-image (61% success rate) and feature-tracks (55% success rate) controller baselines. Furthermore, visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.

PencilNet: Zero-Shot Sim-to-Real Transfer Learning for Robust Gate Perception in Autonomous Drone Racing

“…Loquercio et al [20] train DroNet with simulation data using domain randomization and also change the output layers to produce a velocity vector that drives the robot through the gate. Pfeiffer et al [22] combine DroNet with an attention map as an end-to-end network to capture visual-spatial information allowing the system to track high-speed trajectories. In this work, we compare PencilNet with two variants of DroNet: DroNet-1.0 with the full set of parameters as in [17], and DroNet-0.5 with only half of the filters as in [20].…”

“…The approach in [15] also segments the gates with U-Net but associates the gate corners differently by combining corner pixel searching and re-projection error rejection. Recently, end-to-end methods [20]- [22] using a single DNN that outputs tracking targets for controllers are considered to reduce the latency introduced by traditional modulation of perception, planning, and control sub-problems. In both lines of work, the quality of a gate perception, either through explicit mapping of gates or implicit feature-based understanding, dictates the performance of the system overall.…”

Autonomous Drone Racing: A Survey

“…In addition, eliminating inertial information might have some engineering advantages too, e.g., data throughput, power consumption, and lower cost. Seminal works in this direction try to understand how humans solve this task [6], [171]. They found that expert pilots can control drones despite a 200ms latency, which is compensated by the human brain.…”

Unmanned aerial vehicle navigation in underground structure inspection: A review

“…The robotics and perception (RPG) and autonomous system lab (ASL) teams at the University of Zürich (UZH) both focus on the improvement of navigation methods for visual perception (Pfeiffer et al, 2022). They make Agilicious an open‐source framework for quadrotor drones (Foehn et al, 2022).…”