Double-Prong ConvLSTM for Spatiotemporal Occupancy Prediction in Dynamic Environments

Abstract

Predicting the future occupancy state of an environment is important to enable informed decisions for autonomous vehicles. Common challenges in occupancy prediction include vanishing dynamic objects and blurred predictions, especially for long prediction horizons. In this work, we propose a double-prong neural network architecture to predict the spatiotemporal evolution of the environment occupancy state. One prong is dedicated to predicting how the static environment will be observed by the moving ego vehicle. The other prong predicts how the dynamic objects in the environment will move. Experiments conducted on the real-world Waymo Open Dataset indicate that the fused output of the two prongs is capable of retaining dynamic objects and reducing blurriness in the predictions for longer time horizons than baseline models.

Dynamics-Aware Spatiotemporal Occupancy Prediction in Urban Environments

“…Vanishing moving objects in the predictions, especially at longer prediction time horizons, are common occurrences [7], [11]- [13], [18], [19]. A recent attempt to counter the disappearance problem works by directly integrating dynamic information of the environment into a double-prong model, based on the PredNet architecture [3], by means of separating the OGM inputs into globally static OGMs and dynamic OGMs, and feeding these into the static, and dynamic prong, respectively [11], [12].…”

“…the predictions, but the approach suffers from moving object disappearance in the predictions at longer prediction time horizons. Subsequent attempts to prevent vanishing objects work by directly incorporating environment dynamics into the model in the form of a double-prong architecture [11], [12], or by augmenting with attention mechanism [13].…”

“…Inspired by the success of incorporating environment dynamics directly into the model [11], [12], we develop the future occupancy prediction model that incorporates environment semantics. The model receives as input the history of both the past occupancy information (OGMs), as well as the past semantic information in the form of semantic grid maps (SMGMs), and predicts the future occupancy states in the environment, as illustrated in Fig.…”

LOPR: Latent Occupancy PRediction using Generative Models

“…Lange et al [3] reduced the blurring and the gradual disappearance of dynamic obstacles in the predicted grids by developing an attention augmented ConvLSTM mechanism. Concurrently, Toyungyernsub et al [2] addressed obstacle disappearance with a double-prong framework assuming knowledge of the static and dynamic obstacles present in the scene. Predicted occupancy grids often lack agent identity information.…”

SeqOT: A Spatial-Temporal Transformer Network for Place Recognition Using Sequential LiDAR Data

“…The multi-scan transformer module operates on the concatenated features from consecutive three observations focusing on the relations of temporal features. Different from existing spatial-temporal networks [9], [38], [39] using convolution networks on multiple scans, we use two transformers at different scales to capture both local information of a single scan and sequential relations between multiple scans. We apply the multi-scan transformer on three consecutive scans to exploit spatial-temporal information while keeping the network lightweight and efficient.…”