Is attention to bounding boxes all you need for pedestrian action prediction?

Abstract

The human driver is no longer the only one concerned with the complexity of the driving scenarios. Autonomous vehicles (AV) are similarly becoming involved in the process. Nowadays, the development of AV in urban places underpins essential safety concerns for vulnerable road users (VRUs) such as pedestrians. Therefore, to make the roads safer, it is critical to classify and predict their future behavior. In this paper, we present a framework based on multiple variations of the Transformer models to reason attentively about the dynamic evolution of the pedestrians' past trajectory and predict its future actions of crossing or not crossing the street. We proved that using only bounding-boxes as input to our model can outperform the previous state-of-the-art models and reach a prediction accuracy of 91% and an F1-score of 0.83 on the PIE dataset up to two seconds ahead in the future. In addition, we introduced a large-size simulated dataset (CP2A) using CARLA for action prediction. Our model has similarly reached high accuracy (91 %) and F1-score (0.91) on this dataset. Interestingly, we showed that pre-training our Transformer model on the simulated dataset and then fine-tuning it on the real dataset can be very effective for the action prediction task.

Assessing Cross-dataset Generalization of Pedestrian Crossing Predictors

“…More successful approaches were designed to take into account temporal coherence in short-term motions of visual features of the pedestrians by using ConvLSTMs [10], [11], 3D Convolutions [12], [13], [14], or Spatio-Temporal DenseNet [15]. Approaches trying to minimize the inference time of their models by avoiding the usage of RGB images were explored: [16] proposes a transformer using only spatial positioning of the pedestrian based on 2D bounding box locations. Crossing prediction based on kinematics only was also explored with various available learning architectures to monitor temporal evolution of skeletal joints such as convolutions [17], [18], [19], recurrent cells [20], [21] or graph-based models [22].…”

“…In its first year of existence, proposed approaches evaluated on the benchmarks [1] constantly report higher classification scores [19], [16], [26], [29], [30], [31], giving the impression of clear improvements in pedestrian intention prediction. Usually, a new algorithm is proposed and the implicit hypothesis towards the proposed contribution is made such that it yields an improved performance over the existing state-ofthe-art.…”

Pedestrian Crossing Intention Prediction Model Considering Social Interaction between Multi-Pedestrians and Multi-Vehicles

“…Saleh et al employed a spatio-temporal densenet for classification based on sequences of pedestrian bounding boxes (14). Achaji et al incorporated a transformer for classifying pedestrian intentions based on bounding box features (15).…”