Rules of the Road: Predicting Driving Behavior With a Convolutional Model of Semantic Interactions

Abstract

We focus on the problem of predicting future states of entities in complex, real-world driving scenarios. Previous research has used low-level signals to predict short time horizons, and has not addressed how to leverage key assets relied upon heavily by industry self-driving systems:(1) large 3D perception efforts which provide highly accurate 3D states of agents with rich attributes, and (2) detailed and accurate semantic maps of the environment (lanes, traffic lights, crosswalks, etc). We present a unified representation which encodes such high-level semantic information in a spatial grid, allowing the use of deep convolutional models to fuse complex scene context. This enables learning entity-entity and entity-environment interactions with simple, feed-forward computations in each timestep within an overall temporal model of an agent's behavior. We propose different ways of modelling the future as a distribution over future states using standard supervised learning. We introduce a novel dataset providing industry-grade rich perception and semantic inputs, and empirically show we can effectively learn fundamentals of driving behavior.

Human motion trajectory prediction: a survey

“…For predicting motion of both cyclists and vehicles is it important to consider multi-modality and uncertainty of the future motion. Recently many authors have proposed solutions to this end (Chai et al, 2019; Cui et al, 2019; Hong et al, 2019; Zhao et al, 2019). Furthermore, it is important to consider coordination of actions between the vehicles (Rhinehart et al, 2019; Schmerling et al, 2018).…”

“…Recently, several approaches for non-sequential prediction of vehicle motion using CNNs were presented (Cui et al, 2019; Djuric et al, 2018; Hong et al, 2019). An uncertainty-aware CNN-based vehicle motion prediction approach was presented by Djuric et al (2018).…”

“…The approach is extended by Cui et al (2019) to inferring multi-modal predictions. Hong et al (2019) proposed two methods for output representation using multi-modal regression with uncertainty or stacks of grid-map crops. Chai et al (2019) used a fixed set of state-sequence “anchor” trajectories (clustered from training data), which correspond to possible modes of future behavior, as input to a CNN for mid-level scene features inference, and predict a discrete distribution over these anchors.…”

“…Many deep learning methods consider interactions between participants: explicitly modeling interacting entities (Alahi et al, 2016; Amirian et al, 2019; Bartoli et al, 2018; Choi et al, 2019; Eiffert and Sukkarieh, 2019; Fernando et al, 2018, 2019; Gupta et al, 2018; Hasan et al, 2018; Huang et al, 2019; Ivanovic and Pavone, 2019; Kosaraju et al, 2019; Pei et al, 2019; Pfeiffer et al, 2018; Radwan et al, 2018; Rhinehart et al, 2019; Sadeghian et al, 2019; Saleh et al, 2019; Shi et al, 2019; Su et al, 2017; van der Heiden et al, 2019; Varshneya and Srinivasaraghavan, 2017; Vemula et al, 2018; Xu et al, 2018; Xue et al, 2018; Zhao et al, 2019), implicitly as a result of pixel-wise prediction (Walker et al, 2014), or by learning a joint motion policy (Lee et al, 2017; Ma et al, 2017; Shalev-Shwartz et al, 2016; Zhan et al, 2018). Many vehicle prediction methods consider interaction between traffic participants (e.g., Agamennoni et al, 2012; Altché and de La Fortelle, 2017; Bahram et al, 2016; Broadhurst et al, 2005; Chai et al, 2019; Cui et al, 2019; Dai et al, 2019; Deo and Trivedi, 2018; Ding et al, 2019; Djuric et al, 2018; Hong et al, 2019; Jain et al, 2019; Käfer et al, 2010; Kim et al, 2017; Kuhnt et al, 2016; Li et al, 2019; Park et al, 2018; Raipuria et a...…”

“…Owing to the high level of structure in the environment, methods in autonomous driving scenarios extensively use available semantic information, such as street layout and traffic rules (Agamennoni et al, 2012; Chai et al, 2019; Choi and Savarese, 2010; Cui et al, 2019; Djuric et al, 2018; Gu et al, 2016; Hong et al, 2019; Jain et al, 2019; Keller and Gavrila, 2014; Kooij et al, 2014; Kuhnt et al, 2016; Lee et al, 2017; Petrich et al, 2013; Pool et al, 2019, 2017; Srikanth et al, 2019; Xie et al, 2018) or current state of the traffic lights (Gu et al, 2016; Jain et al, 2019; Karasev et al, 2016), also for predicting pedestrian and cyclist motion (Habibi et al, 2018; Kooij et al, 2019; Koschi et al, 2018).…”