FlightGoggles: Photorealistic Sensor Simulation for Perception-driven Robotics using Photogrammetry and Virtual Reality

Abstract

FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it also provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in flight in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s) in flight. While a vehicle is in flight in the Flight-Goggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex extrinsic dynamics are generated organically through the natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehiclein-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. Flight-Goggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest.

The Blackbird UAV dataset

“…In addition to the rendering challenges described, the rendered imagery will only ever be as good as the underlying models used. While generating high-quality realistic models optimized for computer vision on a scale useful for robotics is possible (see, for example, Guerra et al, 2019a), it still requires a significant amount of artist effort and time to generate. The models used in this dataset were carefully selected for their photorealistic quality, and further modified to provide the best computer vision results.…”

“…Visual data was generated in post-process using the FlightGoggles photo realistic image-generation system from Sayre-McCord et al (2018) and Guerra et al (2019a). FlightGoggles uses the ground-truth 6D pose of the drone from motion-capture to generate images from the viewpoint of each camera on the drone in a virtual environment.…”

“…The goal of the challenge is to incentivize developing algorithms to beat an expert human pilot. For the simulation phase of the challenge, the FlightGoggles simulator presented by Guerra et al (2019a) was used. The exteroceptive sensor data presented in this dataset is generated using the same simulator.…”

Aggressive Perception-Aware Navigation Using Deep Optical Flow Dynamics and PixelMPC

“…Indeed, the quadrotor in the FlightGoggles takes U as an input and the low-level PID controller controls the robot to follow the commands. Since we directly input the angular rates, we do not use the dynamics of the angular rates, described in [20] when we propagate the model in MPC.…”

“…The race course in the FlightGoggles[20] used in this paper. Image credit: the AlphaPilot-Lockheed Martin AI Drone Racing Innovation Challenge 1 .…”