3D visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection

Abstract

Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mapping, visual localization, obstacle detection, etc. need to be adapted to take full advantage of the availability of multiple cameras rather than treat each camera individually. In addition, processing of fisheye images has to be supported. In this paper, we describe the camera calibration and subsequent processing pipeline for multi-fisheye-camera systems developed as part of the V-Charge project. This project seeks to enable automated valet parking for self-driving cars. Our pipeline is able to precisely calibrate multi-camera systems, build sparse 3D maps for visual navigation, visually localize the car with respect to these maps, generate accurate dense maps, as well as detect obstacles based on real-time depth map extraction.

Controlling three-dimensional optical fields via inverse Mie scattering

“…Such an element creating lens-like PSF that varies predictably with different values of defocus would find applications in imaging systems concerned with extracting features at specific depth planes without performing a deconvolution operation. Some example applications of the reported device include depth-sensing cameras for autonomous navigation (40) and bright-field imaging of larger biological structures (26). The demonstrated optical elements have no analog in traditional optics and are impractical to design via intuition.…”

Humanlike Driving: Empirical Decision-Making System for Autonomous Vehicles

“…Fortunately, this is a fundamental task in robotic and self-driving systems, and many effective approaches have been proposed, e.g. [4]- [6].…”

Lane-Level Road Network Generation Techniques for Lane-Level Maps of Autonomous Vehicles: A Survey

“…A digital camera sensor provides digital images where image information such as color, intensity, and texture can be used to detect objects in the scene. Since cameras are low-cost compared to other sensors such as laser scanners, it has been a research hotspot for lane extraction [38,39]. Sensors for vision-based studies include single cameras and stereo cameras [40].…”