STI-GAN: Multimodal Pedestrian Trajectory Prediction Using Spatiotemporal Interactions and a Generative Adversarial Network

Huang, Lei; Zhuang, Jihui; Cheng, Xiaoming; Xu, Riming; Ma, Hongjie

doi:10.1109/access.2021.3069134

Cited by 22 publications

(16 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Narrowing down on the GAN context specifically, we find that spatio-temporal applications have mostly focused on video data (Xu et al 2020;Tulyakov et al 2018;Kim, Oh, and Kim 2020). Beyond this, GANs have been used for conditional density estimation of traffic (Zhang et al 2020), trajectory prediction (Huang et al 2021) or extreme weather event simulation . Nevertheless, to the best of our knowledge, metrics capturing spatio-temporal autocorrelation have never been integrated into GANs.…”

Section: Deep Learning and Gans For Spatial And Spatio-temporal Datamentioning

confidence: 99%

SPATE-GAN: Improved Generative Modeling of Dynamic Spatio-Temporal Patterns with an Autoregressive Embedding Loss

Klemmer

Acciaio

et al. 2022

AAAI

View full text Add to dashboard Cite

From ecology to atmospheric sciences, many academic disciplines deal with data characterized by intricate spatio-temporal complexities, the modeling of which often requires specialized approaches. Generative models of these data are of particular interest, as they enable a range of impactful downstream applications like simulation or creating synthetic training data. Recently, COT-GAN, a new GAN algorithm inspired by the theory of causal optimal transport (COT), was proposed in an attempt to improve generation of sequential data. However, the task of learning complex patterns over time and space requires additional knowledge of the specific data structures. In this study, we propose a novel loss objective combined with COT-GAN based on an autoregressive embedding to reinforce the learning of spatio-temporal dynamics. We devise SPATE (spatio-temporal association), a new metric measuring spatio-temporal autocorrelation. We compute SPATE for real and synthetic data samples and use it to compute an embedding loss that considers space-time interactions, nudging the GAN to learn outputs that are faithful to the observed dynamics. We test our new SPATE-GAN on a diverse set of spatio-temporal patterns: turbulent flows, log-Gaussian Cox processes and global weather data. We show that our novel embedding loss improves performance without any changes to the architecture of the GAN backbone, highlighting our model's increased capacity for capturing autoregressive structures.

show abstract

Section: Deep Learning and Gans For Spatial And Spatio-temporal Datamentioning

confidence: 99%

SPATE-GAN: Improved Generative Modeling of Dynamic Spatio-Temporal Patterns with an Autoregressive Embedding Loss

Klemmer

Acciaio

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Considering the deficiencies of Social-GAN, Psalta et al [14] introduce Edge Convolution Pooling (ECP) to replace Social Pooling with the same framework of Social-GAN, but ECP captures fixed K neighbors around the target which may get some information lost. Huang et al [15] propose STI-GAN which also with a GAT module embedded into the GAN framework. STI-GAN successfully captures and aggregates Spatio-temporal features, but fails to consider the unequal importance of different observed moments.…”

Section: B Gan For Pedestrian Trajectory Predictionmentioning

confidence: 99%

An Enhanced Representation Method for Pedestrian Trajectory Prediction based on Adaptive GCN

Zhang

Jiang

Hui

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

View full text Add to dashboard Cite

Recently, recurrent neural networks, like Long Short-Term Memory (LSTM) networks, have been widely used for sequence prediction within complex scenes [3]-[5], and the unique memory units in the network which help memorize historical continuous features have a rather effective performance in the mitigating error propagation. But they have a weak performance on social interaction modeling. Based on the strength of RNNs, an alternative kind of method, i.e. Generative Adversarial Networks (GANs) based methods, is proposed to model the uncertainty of pedestrian trajectories caused by potential factors in real scenes [6], [7]. However, GANs-based methods mostly address each trajectory separately on interaction modeling, which ignores some key factors within social interactions and has a high computational cost. Currently, graph and graph neural network-based methods [8]-[10] are widely used, because graph structures are more intuitive and understandable in modeling physical and social interactions among pedestrians. However, most of them still suffer from limitations when coping with social interactions, as some methods default to the fixed connection among all pedestrians while some others sample a fixed number of neighbors to build the graph. Thus, the approaches either introduce much noise or loss some implicit information, like time, when modeling social interactions, while the problem with error accumulation does not get sufficient consideration in these approaches, which results in the suboptimal predictions.

show abstract

“…SoPhie encourages acquiring social interactions from the pedestrian through a social attention mechanism. Huang et al 41 proposed an attentive group‐aware GAN to observe the agents' past motion and predict future paths. Amirian et al 17 used InfoGAN 42 to perform unsupervised learning based on data with potential categories.…”

Section: Related Workmentioning

confidence: 99%

An improved GAN with transformers for pedestrian trajectory prediction models

Huang

Cao

2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

Predicting the future trajectories of multiple pedestrians in certain scenes is critical for autonomous moving platforms (like, self-driving cars and social robots). In this paper, we propose a novel Generative Adversarial Network model with Transformers, which simulates the pedestrian distribution to capture the uncertainty of the predicted paths and generate more reasonable future trajectories. The design of our method includes a generator and a discriminator. The generator mainly contains an encoder, a decoder, and a prediction module. Specifically, the encoder and the decoder comprise multihead convolutional selfattention to learn the sequence of historical movement, and the prediction module incorporates the Mish Feed-Forward Network to yield the predicted target. The discriminator takes both the predicted paths and ground truth as input, classifies them as socially acceptable or not. Experimental results show that the proposed method consistently boosts the performance of trajectory forecasting, and our framework surpasses several existing baselines by evaluating the results on various data sets. Code is available at https://github. com/lzz970818/Trajectory-Prediction.

show abstract

STI-GAN: Multimodal Pedestrian Trajectory Prediction Using Spatiotemporal Interactions and a Generative Adversarial Network

Cited by 22 publications

References 27 publications

SPATE-GAN: Improved Generative Modeling of Dynamic Spatio-Temporal Patterns with an Autoregressive Embedding Loss

SPATE-GAN: Improved Generative Modeling of Dynamic Spatio-Temporal Patterns with an Autoregressive Embedding Loss

An Enhanced Representation Method for Pedestrian Trajectory Prediction based on Adaptive GCN

An improved GAN with transformers for pedestrian trajectory prediction models

Contact Info

Product

Resources

About