CenterLoc3D: monocular 3D vehicle localization network for roadside surveillance cameras

Abstract

Monocular 3D vehicle localization is an important task for vehicle behaviour analysis, traffic flow parameter estimation and autonomous driving in Intelligent Transportation System (ITS) and Cooperative Vehicle Infrastructure System (CVIS), which is usually achieved by monocular 3D vehicle detection. However, monocular cameras cannot obtain depth information directly due to the inherent imaging mechanism, resulting in more challenging monocular 3D tasks. Currently, most of the monocular 3D vehicle detection methods still rely on 2D detectors and additional geometric constraint modules to recover 3D vehicle information, which reduces the efficiency. At the same time, most of the research is based on datasets of onboard scenes, instead of roadside perspective, which is limited in large-scale 3D perception. Therefore, we focus on 3D vehicle detection without 2D detectors in roadside scenes. We propose a 3D vehicle localization network CenterLoc3D for roadside monocular cameras, which directly predicts centroid and eight vertexes in image space, and the dimension of 3D bounding boxes without 2D detectors. To improve the precision of 3D vehicle localization, we propose a multi-scale weighted-fusion module and a loss with spatial constraints embedded in CenterLoc3D. Firstly, the transformation matrix between 2D image space and 3D world space is solved by camera calibration. Secondly, vehicle type, centroid, eight vertexes, and the dimension of 3D vehicle bounding boxes are obtained by CenterLoc3D. Finally, centroid in 3D world space can be obtained by camera calibration and CenterLoc3D for 3D vehicle localization. To the best of our knowledge, this is the first application of 3D vehicle localization for roadside monocular cameras. Hence, we also propose a benchmark for this application including a dataset (SVLD-3D), an annotation tool (LabelImg-3D), and evaluation metrics. Through experimental validation, the proposed method achieves high accuracy with $$A{P_{3D}}$$ A P 3 D of 51.30%, average 3D localization precision of 98%, average 3D dimension precision of 85% and real-time performance with FPS of 41.18.

Experimental Study of Multi-Camera Infrastructure Perception for V2X-Assisted Automated Driving in Highway Merging

“…To begin with, the localisation accuracy is improved compared to [28], where the localisation error varies between -2m and 4m, despite higher resolution images and knowing the camera extrinsics. In addition, our localisation accuracy is on par to [29], that reports a mean error of 33cm (compared to ours of 48cm). However, the localisation error in [29] increases dramatically when the vehicle to camera distance is greater than 80m, with maximum errors going above 1m at distances approaching 100m, despite also knowing the camera extrinsics.…”

“…In addition, our localisation accuracy is on par to [29], that reports a mean error of 33cm (compared to ours of 48cm). However, the localisation error in [29] increases dramatically when the vehicle to camera distance is greater than 80m, with maximum errors going above 1m at distances approaching 100m, despite also knowing the camera extrinsics. The physical infrastructure used in our study had a distance separation between infrastructure poles (each consisting of 2 cameras) of roughly 125m.…”

“…Existing monocular multi-camera fusion solutions often assume knowledge of the extrinsic/intrinsic camera transformations [27]- [29]. Moreover, since the camera perspective from infrastructure sensors is vastly different, depth-estimating approaches designed for onboard sensors are either inappropriate or require retraining/recreating the depth-density map [30].…”

A discrete-time distributed optimization algorithm for cooperative transportation of multi-robot system

“…For example, MRS performs the task in a distributed manner, and enhances the robustness and fault tolerance of the system. Therefore, MRS has been applied to many engineering problems, such as formation control [2,3], cooperative transportation [4,5] and cooperative manufacture [6,7].…”