Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

Gesnouin, Joseph; Pechberti, Steve; Bresson, Guillaume; Stanciulescu, Bogdan; Moutarde, Fabien

doi:10.3390/a13120331

Cited by 26 publications

(13 citation statements)

References 87 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [ 11 ], pedestrian 2D pose keypoints sequence from a specific time interval (≈0.5 s) is used as an input to classify, with a random forest model, the pedestrian crossing action being performed. In [ 12 ], precomputed features extracted from the complete sequence of pose keypoints are used as input for a multi-branch 2D CNN network. In [ 13 ], whole sequences of pedestrian’s keypoints are used in a similar way to the previous one, but generating adjacency matrix representations based on the pose graph, as an approximation to graph learning.…”

Section: Related Workmentioning

confidence: 99%

CAPformer: Pedestrian Crossing Action Prediction Using Transformer

Lorenzo

Alonso

Izquierdo

et al. 2021

Sensors

View full text Add to dashboard Cite

Anticipating pedestrian crossing behavior in urban scenarios is a challenging task for autonomous vehicles. Early this year, a benchmark comprising JAAD and PIE datasets have been released. In the benchmark, several state-of-the-art methods have been ranked. However, most of the ranked temporal models rely on recurrent architectures. In our case, we propose, as far as we are concerned, the first self-attention alternative, based on transformer architecture, which has had enormous success in natural language processing (NLP) and recently in computer vision. Our architecture is composed of various branches which fuse video and kinematic data. The video branch is based on two possible architectures: RubiksNet and TimeSformer. The kinematic branch is based on different configurations of transformer encoder. Several experiments have been performed mainly focusing on pre-processing input data, highlighting problems with two kinematic data sources: pose keypoints and ego-vehicle speed. Our proposed model results are comparable to PCPA, the best performing model in the benchmark reaching an F1 Score of nearly 0.78 against 0.77. Furthermore, by using only bounding box coordinates and image data, our model surpasses PCPA by a larger margin (F1=0.75 vs. F1=0.72). Our model has proven to be a valid alternative to recurrent architectures, providing advantages such as parallelization and whole sequence processing, learning relationships between samples not possible with recurrent architectures.

show abstract

Section: Related Workmentioning

confidence: 99%

CAPformer: Pedestrian Crossing Action Prediction Using Transformer

Lorenzo

Alonso

Izquierdo

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…The Joint Attention in Autonomous Driving (JAAD) dataset [110] was used in the research studies [12], [51], and [71]. This dataset focuses on pedestrian and driver activity at crossings and the factors that affect it.…”

Section: Figure 2challenges For Behavior Prediction Of Traffic Actors Based On Input Representation Rq 3 What Are the Different Datasets mentioning

confidence: 99%

“…Dataset Name Application and Purpose Feature [12], [51], [71] Joint Attention in Autonomous Driving (JAAD) dataset…”

Section: Referencementioning

confidence: 99%

Behavior Prediction of Traffic Actors for Intelligent Vehicle Using Artificial Intelligence Techniques: A Review

Kolekar

Gite²,

Pradhan

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Intelligent vehicle technology has made tremendous progress due to Artificial Intelligence (AI) techniques.Accurate behavior prediction of surrounding traffic actors is essential for the safe and secure navigation of the intelligent vehicle. Minor misbehavior of these vehicles on the busy roads may lead to an accident. Due to this, there is a need for vehicle behavior research work in today's era. This research article reviews traffic actors' behavior prediction techniques for intelligent vehicles to perceive, infer, and anticipate other vehicles' intentions and future actions. It identifies the key strategies and methods for AI, emerging trends, datasets, and ongoing research issues in these fields. As per the authors' knowledge, this is the first systematic literature review dedicated to the vehicle behavior study examining existing academic literature published by peer review venues between 2011 and 2021. A systematic review was undertaken to examine these papers, and five primary research questions have been addressed. The findings show that using sophisticated input representation that includes traffic rules and road geometry, artificial intelligence-based solutions applied to behavior prediction of traffic actors for intelligent vehicles have shown promising success, particularly in complex driving scenarios. Finally, the paper summarizes the most widely used approaches in behavior prediction of traffic actors for intelligent vehicles, which the authors believe serves as a foundation for future research in behavior prediction of surrounding traffic actors for secure and accurate intelligent vehicle navigation. INDEX TERMS intelligent driving; deep learning; intelligent vehicles; vehicle behavior prediction; pedestrian behavior prediction. Inclusion criteriaResearch articles must be original research articles instead of review/survey articles. Research articles that are published between2011 and 2021.Research articles that answer at least one research question. Research articles must contain search keywords either in the title, abstract, or full text. The proposed solution in the research article must be evaluated. The solution developed must be aimed to solve problems on behavior prediction of traffic actors. Exclusion criteriaNon-English Research articles Duplicate research articles Research articles with non-availability of full text Research articles that are not relevant to pedestrians and vehicle behavior prediction concerning intelligent vehicle III. RESULTSThis section summarizes the findings of systematic analysis. It answers the research questions posed above based on the results of this review process.

show abstract

“…This breakthrough has stimulated the skeletal modality interest since it proved to be sufficient to describe and understand the motion of a given action without any background context. This has made pose-based action recognition preferred over other modalities on a huge amount of real-time scenarios for human action recognition such as human-robot interaction [24], [3], medical rehabilitative applications [25], [8] or pedestrian action prediction [12], [11], [13]. Some commonly used learning architectures for pose-based action recognition include 1D/2D convolutional networks [9], [27], recurrent networks [1], [39], a combination of one of the latter with attention mechanisms [21], [16] or Graph-based models [56], [50].…”

Section: A Pose-based Action Recognitionmentioning

confidence: 99%

“…All those methods present a drawback: they become sensitive to noise, background, and illumination conditions by including scene images in their approaches. To overcome these issues, intention prediction only based on 2D body poses sequences has been explored with various available learning architectures such as convolutions [11], recurrent cells [22], [13], graph-based models [4] and proposed to enhance pose-based approaches by creating features based on body structure to capture different aspects of the data [31], [12]. However, the lack of a common evaluation criterion, of normalized modalities inputs, of a common observation frames selection method, and common prediction horizons made the task of comparing each approach's robustness difficult if not impossible to realize.…”

Section: B Pedestrian Action Predictionmentioning

confidence: 99%

TrouSPI-Net: Spatio-temporal attention on parallel atrous convolutions and U-GRUs for skeletal pedestrian crossing prediction

Gesnouin

Pechberti

Stanciulescu

et al. 2021

2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)

Self Cite

View full text Add to dashboard Cite

Understanding the behaviors and intentions of pedestrians is still one of the main challenges for vehicle autonomy, as accurate predictions of their intentions can guarantee their safety and driving comfort of vehicles. In this paper, we address pedestrian crossing prediction in urban traffic environments by linking the dynamics of a pedestrian's skeleton to a binary crossing intention. We introduce TrouSPI-Net: a context-free, lightweight, multi-branch predictor. TrouSPI-Net extracts spatio-temporal features for different time resolutions by encoding pseudo-images sequences of skeletal joints' positions and processes them with parallel attention modules and atrous convolutions. The proposed approach is then enhanced by processing features such as relative distances of skeletal joints, bounding box positions, or ego-vehicle speed with U-GRUs. Using the newly proposed evaluation procedures for two large public naturalistic data sets for studying pedestrian behavior in traffic: JAAD and PIE, we evaluate TrouSPI-Net and analyze its performance. Experimental results show that TrouSPI-Net achieved 76% F1 score on JAAD and 80% F1 score on PIE, therefore outperforming current state-of-the-art while being lightweight and context-free.

show abstract

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

Cited by 26 publications

References 87 publications

CAPformer: Pedestrian Crossing Action Prediction Using Transformer

CAPformer: Pedestrian Crossing Action Prediction Using Transformer

Behavior Prediction of Traffic Actors for Intelligent Vehicle Using Artificial Intelligence Techniques: A Review

TrouSPI-Net: Spatio-temporal attention on parallel atrous convolutions and U-GRUs for skeletal pedestrian crossing prediction

Contact Info

Product

Resources

About