VeReMi: A Dataset for Comparable Evaluation of Misbehavior Detection in VANETs

Abstract

Vehicular networks are networks of communicating vehicles, a major enabling technology for future cooperative and autonomous driving technologies. The most important messages in these networks are broadcast-authenticated periodic one-hop beacons, used for safety and traffic efficiency applications such as collision avoidance and traffic jam detection. However, broadcast authenticity is not sufficient to guarantee message correctness. The goal of misbehavior detection is to analyze application data and knowledge about physical processes in these cyber-physical systems to detect incorrect messages, enabling local revocation of vehicles transmitting malicious messages. Comparative studies between detection mechanisms are rare due to the lack of a reference dataset. We take the first steps to address this challenge by introducing the Vehicular Reference Misbehavior Dataset (VeReMi) and a discussion of valid metrics for such an assessment. VeReMi is the first public extensible dataset, allowing anyone to reproduce the generation process, as well as contribute attacks and use the data to compare new detection mechanisms against existing ones. The result of our analysis shows that the acceptance range threshold and the simple speed check are complementary mechanisms that detect different attacks. This supports the intuitive notion that fusion can lead to better results with data, and we suggest that future work should focus on effective fusion with VeReMi as an evaluation baseline.

CaTch: A Confidence Range Tolerant Misbehavior Detection Approach

“…The attacker generates an identity and manages a correct transmission frequency for each ghost vehicle. The Constant Offset is a type of misbehavior often used in this field [7] [9], it is more a simulation of a faulty device than an attack. As for the Sybil Attack it is a more complicated type of misbehavior that could cause more disruption to the system.…”

“…Using this partition we can calculate, similarly to other studies [7] [16], the following performance metrics: Intuitively, the Recall characterizes the system ability to flag all the misbehaving messages, Whereas the P recision characterizes the system ability to not consider misbehaving as genuine messages. The F 1 score is the harmonic mean of Recall and P recision.…”

Shadows Don't Lie: n-Sequence Trajectory Inspection for Misbehaviour Detection and Classification in VANETs

“…The VeReMi dataset has recently been published and is built specifically for testing location spoofing attacks in V2X scenarios [5]. The advantages of using the VeReMi dataset are (i) it presents different types of attackers; (ii) it simulates varied density conditions of vehicle network; (iii) it is consistent with its BSM's broadcast rate; and (iv) the scenarios are conveniently labelled for using a machine learning approach.…”

“…We develop machine learning models (K-Nearest Neighbour and Support Vector Machine, KNN and SVM, respectively) based on three features which can be extracted from consecutive location observations in the trajectory data: Movement Plausibility Check, Minimum Trajectory Distance, and Minimum Translation Trajectory Distance. We evaluate the performance of our models in detecting and classifying attacks in the VeReMi dataset [5], a labelled dataset built in VEINS [6], which offers five different types of location spoofing attacks (see Table I). We compare our solution to a similar machine learning approach presented in [1].…”

Towards a Reliable Machine Learning-Based Global Misbehavior Detection in C–ITS: Model Evaluation Approach

“…They build two machine learning models: k-nearest neighbors (k-NN) and support vector machine (SVM). The training and data evaluation are done on using VeReMi dataset [9]. They show that they can improve the overall detection precision of the plausibility checks used in the feature vectors by over 20%, while maintaining a recall within 5% of that of the plausibility checks.…”