MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses

Abstract

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.

Looking to Relations for Future Trajectory Forecast

“…To predict a future trajectory of pedestrians from first-person videos, temporal changes of orientation and body pose are encoded as one of the features in [45]. In parallel, [13] uses head pose as a proxy to build a better forecasting model. Both methods find that gaze, inferred by the body or head orientation, and Figure 2: Given a sequence of images, the GRE visually analyzes spatial behavior of road users and their temporal interactions with respect to environments.…”

“…Such technologies require advanced decision making and motion planning systems that rely on estimates of the future position of road users in order to realize safe and effective mitigation and navigation strategies. Related research [46,1,36,23,37,12,13,43,45,32,33,47] has attempted to predict future trajectories by focusing on social conventions, environmental factors, or pose and motion constraints. They have shown to be more effective when the prediction model learns to extract these features by considering human-human (i.e., between road agents) or human-space (i.e., between a road agent and environment) interactions.…”

Scene Compliant Trajectory Forecast With Agent-Centric Spatio-Temporal Grids

“…Alternatively, some approaches use temporal convolutional networks for encoding sequences of past locations [12], [13], allowing for faster run-times. In addition to location co-ordinates, some approaches also incorporate auxiliary information such as the head pose of pedestrians [9], [14] while encoding past motion. Many approaches jointly model the past motion of multiple agents in the scene to capture interaction between agents [5], [15], [12], [10], [7], [11].…”

Effects of a Social Force Model Reward in Robot Navigation Based on Deep Reinforcement Learning

“…Supervised Learning techniques have been applied to predict the movements of agents in a temporal horizon. For example, sequence models that use Long Short Term Memory (LSTM) recurrent neural networks like Social LSTM [1] and other [12,13] are capable to encode the Human-Robot interactions and Human-Human interactions to improve the predictions. Other techniques are based in generative models like Social-GAN [11] or SoPhie [21] that use pooling modules and attention modules.…”