Large Batch Simulation for Deep Reinforcement Learning

Abstract

We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of over 19,000 frames of experience per second on a single GPU and up to 72,000 frames per second on a single eight-GPU machine. The key idea of our approach is to design a 3D renderer and embodied navigation simulator around the principle of "batch simulation": accepting and executing large batches of requests simultaneously. Beyond exposing large amounts of work at once, batch simulation allows implementations to amortize in-memory storage of scene assets, rendering work, data loading, and synchronization costs across many simulation requests, dramatically improving the number of simulated agents per GPU and overall simulation throughput. To balance DNN inference and training costs with faster simulation, we also build a computationally efficient policy DNN that maintains high task performance, and modify training algorithms to maintain sample efficiency when training with large mini-batches. By combining batch simulation and DNN performance optimizations, we demonstrate that PointGoal navigation agents can be trained in complex 3D environments on a single GPU in 1.5 days to 97% of the accuracy of agents trained on a prior state-of-the-art system using a 64-GPU cluster over three days. We provide open-source reference implementations of our batch 3D renderer and simulator to facilitate incorporation of these ideas into RL systems.

Landmark Policy Optimization for Object Navigation Task

“…We decided to use the Mat-terport3D dataset because of its size and diversity and the Habitat environment because of its rendering speed and straightforward way to multithread. To train PointNav policy that does not require a semantic sensor, we used the faster version of Habitat (that does not support native semantic segmentation sensor), the BPS simulator (Shacklett et al 2021), which was 100x timed faster. With a similar approach, these works (Kadian et al 2020) (Watkins-Valls et al 2020 showed the ability to transfer the RL model from a simulated environment to real-world usage.…”

Simulation and application of COVID-19 compartment model using physics-informed neural network

“…pytorch [10]) and hardware (e.g. RTX3090 [15]) and the advent of the era of big data, different kinds of Informed Neural Network (INN) has received a lot of research and gradually developed into an important tool to solve many physics, statistics and engineering problems [11]. Different from pure data-driven model or pure knowledge driven model, it can be used as a "bridge" to connect data methods with traditional knowledge driven equations.…”