VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles

Abstract

Simulation has the potential to transform the development of robust algorithms for mobile agents deployed in safety-critical scenarios. However, the poor photorealism and lack of diverse sensor modalities of existing simulation engines remain key hurdles towards realizing this potential. Here, we present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras, enabling the rapid generation of novel viewpoints in simulation and thereby enriching the data available for policy learning with corner cases that are difficult to capture in the physical world. Using VISTA, we demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle. The policies learned in VISTA exhibit sim-to-real transfer without modification and greater robustness than those trained exclusively on real-world data.

Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Driving

“…We start by introducing the hardware platform and data collection, followed by implementation details of the proposed model. We then demonstrate extensive analysis in the sim-to-real environment VISTA [4]. Finally, we showcase results with real-car deployment.…”

“…Works that learn object avoidance in simulation have leveraged both imitation learning [54] as well as reinforcement learning [8,28,11,58,19] but often face limited to no deployment capabilities in reality due to large sim-to-real gaps present in model-based simulation. In this work, we leverage recent advances in data-driven simulation [3,30,46,4] to overcome the sim-to-real gap to learn robust end-to-end controllers capable of transferring to real scenarios with other agents.…”

“…Also, we use a differential global positioning system (dGPS) for evaluation purpose. To run the data-driven simulation VISTA [4], we collect real-world data from a wide-range of environments, including different time of day, weather conditions, and seasons of a year. The entire dataset consists of roughly 2 hour of driving data, which is further augmented with our training dataset generation pipeline using VISTA.…”

“…Open-loop control error (i.e., difference between predicted and ground-truth control) has been shown to be a poor indicator to evaluate the performance of a driving policy since it only measures error around ground-truth trajectories and ignores accumulated errors that gradually drift the vehicle to out-of-distribution regions. Hereby, we presents closed-loop testing results in the sim-to-real environment VISTA [4]. Lane Keeping As Safety Constraints.…”

Revisiting the Adversarial Robustness-Accuracy Tradeoff in Robot Learning

“…2. Test conditions of our closed-loop driving experiment using a data-driven simulation environment [37]. The training data are collected in summer and winter conditions (separated from the testing data).…”