Behavior-Aware Pedestrian Trajectory Prediction in Ego-Centric Camera Views with Spatio-Temporal Ego-Motion Estimation

Abstract

With the ongoing development of automated driving systems, the crucial task of predicting pedestrian behavior is attracting growing attention. The prediction of future pedestrian trajectories from the ego-vehicle camera perspective is particularly challenging due to the dynamically changing scene. Therefore, we present Behavior-Aware Pedestrian Trajectory Prediction (BA-PTP), a novel approach to pedestrian trajectory prediction for ego-centric camera views. It incorporates behavioral features extracted from real-world traffic scene observations such as the body and head orientation of pedestrians, as well as their pose, in addition to positional information from body and head bounding boxes. For each input modality, we employed independent encoding streams that are combined through a modality attention mechanism. To account for the ego-motion of the camera in an ego-centric view, we introduced Spatio-Temporal Ego-Motion Module (STEMM), a novel approach to ego-motion prediction. Compared to the related works, it utilizes spatial goal points of the ego-vehicle that are sampled from its intended route. We experimentally validated the effectiveness of our approach using two datasets for pedestrian behavior prediction in urban traffic scenes. Based on ablation studies, we show the advantages of incorporating different behavioral features for pedestrian trajectory prediction in the image plane. Moreover, we demonstrate the benefit of integrating STEMM into our pedestrian trajectory prediction method, BA-PTP. BA-PTP achieves state-of-the-art performance on the PIE dataset, outperforming prior work by 7% in MSE-1.5 s and CMSE as well as 9% in CFMSE.

VOSTN: Variational One-shot Transformer Network for Pedestrian Trajectory Prediction

“…In general, previous works only make predictions for a single future trajectory, that is, they are deterministic (Rasouli et al 2017, Bhattacharyya et al 2018, Rasouli et al 2019, Czech et al 2023. However, in a real traffic environment, there are multiple future modes for pedestrians.…”

“…The ability to accurately forecast future pedestrian behavior is a relatively simple task for humans, yet it remains a significant challenge for machines at present. Prior research has primarily concentrated on utilizing Recurrent Neural Networks (RNNs) to model (Rasouli et al 2017, Bhattacharyya et al 2018, Rasouli et al 2019, Quan et al 2021, Xue et al 2020, Neumann and Vedaldi 2021, Yao et al 2021, Wang et al 2022, Czech et al 2022, Kalatian and Farooq 2022, Fu and Zhao 2023, Choi et al 2021, Czech et al 2023, with particular emphasis on Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber 1997) or Gate Recurrent Unit (GRU) (Cho et al 2014). Giuliari et al (2021) incorporated the Transformer Network into trajectory prediction due to its outstanding performance in sequence modeling.…”

Holistic Spatio-Temporal Graph Attention for Trajectory Prediction in Vehicle–Pedestrian Interactions

“…The interaction between vehicles and pedestrians is a significant factor in predicting their future trajectories. We build upon the work of [59,60,137,138] and implement a VPI cell into the LSTM to improve trajectory prediction by encoding vehicle-pedestrian interaction features into the individual agent LSTM. The process of extracting features related to vehicle-pedestrian interactions involves two steps, and each step has two stages, as depicted in Figure 4.…”

“…Expanding on this insightful understanding and drawing inspiration from comprehensive studies [59,60,137,138], we propose the integration of an additional memory cell and dynamic rescaling of the output gate in response to changes in vehicle-pedestrian spatial interaction. We have developed a concept termed the "vehicle-pedestrian interaction (VPI) cell" to further augment the intrinsic interactions among these cues.…”