Learn-to-Race: A Multimodal Control Environment for Autonomous Racing

Abstract

Existing research on autonomous driving primarily focuses on urban driving, which is insufficient for characterising the complex driving behaviour underlying highspeed racing. At the same time, existing racing simulation frameworks struggle in capturing realism, with respect to visual rendering, vehicular dynamics, and task objectives, inhibiting the transfer of learning agents to real-world contexts. We introduce a new environment, where agents Learn-to-Race (L2R) in simulated competition-style racing, using multimodal information-from virtual cameras to a comprehensive array of inertial measurement sensors. Our environment, which includes a simulator and an interfacing training framework, accurately models vehicle dynamics and racing conditions. In this paper, we release the Arrival simulator for autonomous racing. Next, we propose the L2R task with challenging metrics, inspired by learning-to-drive challenges, Formula-style racing, and multimodal trajectory prediction for autonomous driving. Additionally, we provide the L2R framework suite, facilitating simulated racing on high-precision models of realworld tracks, such as the famed Thruxton Circuit and the Las Vegas Motor Speedway. Finally, we provide an official L2R task dataset of expert demonstrations, as well as a series of baseline experiments and reference implementations. We make all code available: https://github. com/hermgerm29/learn-to-race

Core Challenges in Embodied Vision-Language Planning

“…Recent works have addressed this lack of realism through the use of photo-realism (Savva et al, 2019;Xia et al, 2018;Anderson et al, 2018c;Herman et al, 2021) and the use of interactive contexts where agents are able to modify the states of objects in the environment (Kolve et al, 2017;Xia et al, 2019;Yan et al, 2018;Brodeur et al, 2018). Toward this end, there is also interest in developing frameworks focused on simulation-to-real transfer and evaluation (Deitke et al, 2020), allowing the study of discrepancies between real settings and simulated ones.…”