Is attention to bounding boxes all you need for pedestrian action prediction?

Abstract

Anticipating human actions in front of autonomous vehicles is a challenging task. Several papers have recently proposed model architectures to address this problem by combining multiple input features to predict pedestrian crossing actions. This paper focuses specifically on using images of the pedestrian's context as an input feature. We present several spatio-temporal model architectures that utilize standard CNN and Transformer modules to serve as a backbone for pedestrian anticipation. However, the objective of this paper is not to surpass state-of-the-art benchmarks but rather to analyze the positive and negative predictions of these models. Therefore, we provide insights on the explainability of visionbased Transformer models in the context of pedestrian action prediction. We will highlight cases where the model can achieve correct quantitative results but falls short in providing humanlike explanations qualitatively, emphasizing the importance of investing in explainability for pedestrian action anticipation problems.

Deep Virtual-to-Real Distillation for Pedestrian Crossing Prediction

“…Because of the rarity of the pedestrian crossing in unmarked road, Achaji et al [4] proposed a new work trained on the large-scale simulated data, and it emphasis that using only bounding boxes of pedestrian can leverage an accurate pedestrian crossing prediction. However, this work does not consider the few-shot problem of the pedestrian crossing problem when encountering severe weather condition, low light conditions.…”

“…Each frame will be automatically marked with five attributes: occasions, weather, gender, age, and the bounding box coordinate of pedestrian. We compare our dataset statistics with JAAD, PIE, and CP2A [4] (recently reported) in terms of sequence numbers, frame numbers, and pedestrian counts, as shown in Table I. Furthermore, since the pedestrian scale has a large impact on the prediction accuracy, we compare the pedestrian scale statistics of our dataset, JAAD and PIE in Fig.…”

“…1. In this field, 2D poses [2], [3], pedestrian bounding boxes [4], optical flow [5], scene context [6], vehicles speeds [7], trajectories [8], ego-motion of vehicles [7] are utilized in previous works. In the meantime, the deep learning models, such as I3D [5], LSTM/RNN-based temporal models [8], [9], as well as the transformers [10] are adopted in recent years.…”

“…However, it is difficult to tackle this problem in practical driving. A recent work [4] explored the virtual data simulated by CARLA [13], which collected a large-scale action prediction dataset and defined the crossing situation. Then, the work took a fine-tuning module to transfer the distribution of synthetic data to the one of real data.…”

“…Then, the work took a fine-tuning module to transfer the distribution of synthetic data to the one of real data. However, in order to make the distribution be transferable, the work [4] only considers the bounding boxes of pedestrians. Nevertheless, bounding boxes have no scene information, and any crossing behavior (not limited to pedestrian crossing) could be treated as pedestrian crossing (false alarm), shown by Fig.…”