RegNet: Multimodal sensor registration using deep neural networks

Schneider, N. M.; Piewak, Florian; Stiller, Christoph; Franke, Uwe

doi:10.1109/ivs.2017.7995968

Cited by 173 publications

(119 citation statements)

References 21 publications

Supporting

Mentioning

117

Contrasting

Unclassified

Order By: Relevance

“…V-A3). Interestingly, several works propose to calibrate sensors by deep neural networks: Giering et al [197] discretize the spatial misalignments between LiDAR and visual camera into nine classes, and build a network to classify misalignment taking LiDAR and RGB images as inputs; Schneider et al [198] propose to fully regress the extrinsic calibration parameters between LiDAR and visual camera by deep learning. Several multi-modal CNN networks are trained on different decalibration ranges to iteratively refine the calibration output.…”

Section: ) Data Quality and Alignmentmentioning

confidence: 99%

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Feng

Haase-Schütz

Rosenbaum

et al. 2021

IEEE Trans. Intell. Transport. Syst.

892

338

View full text Add to dashboard Cite

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of "what to fuse", "when to fuse", and "how to fuse" remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/. 0.99 0.8 0.98 0.99 0.96 0.96 0.94 Vehicle Person Road sign Traffic light LiDAR Points Map Radar Points RGB Image

show abstract

Section: ) Data Quality and Alignmentmentioning

confidence: 99%

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Feng

Haase-Schütz

Rosenbaum

et al. 2021

IEEE Trans. Intell. Transport. Syst.

892

338

View full text Add to dashboard Cite

show abstract

“…The work presented in this paper has been inspired by Schneider et al [26], which used 3D scans from a LiDAR and RGB images as the input of a novel CNN, RegNet. Their goal was to provide a CNN-based method for calibrating the extrinsic parameters of a camera w.r.t.…”

Section: B Camera and Lidar-map Approachesmentioning

confidence: 99%

CMRNet: Camera to LiDAR-Map Registration

Cattaneo¹,

Vaghi²,

Ballardini³

et al. 2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

View full text Add to dashboard Cite

In this paper we present CMRNet, a realtime approach based on a Convolutional Neural Network (CNN) to localize an RGB image of a scene in a map built from LiDAR data. Our network is not trained in the working area, i.e., CMRNet does not learn the map. Instead it learns to match an image to the map. We validate our approach on the KITTI dataset, processing each frame independently without any tracking procedure. CMRNet achieves 0.27m and 1.07 • median localization accuracy on the sequence 00 of the odometry dataset, starting from a rough pose estimate displaced up to 3.5m and 17 • . To the best of our knowledge this is the first CNN-based approach that learns to match images from a monocular camera to a given, preexisting 3D LiDAR-map.

show abstract

“…Yet, surprisingly, only a few deep learning based approaches have been applied to the calibration problem. The first deep [14] regress to the calibration parameters, conditioned on the input image. Since it doesn't take geometry into account, it has to be retrained each time sensor intrinsics change.…”

Section: Related Workmentioning

confidence: 99%

“…For the depth branch, we use a similar architecture as for the RGB stream, but with half the number of filters at each stage. Like in [14], this architecture has several advantages for feature extraction. The use of pre-trained weights for the RGB input prevents learning the relevant features from scratch.…”

Section: (B)mentioning

confidence: 99%

CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks

Iyer

Ram

Murthy

et al. 2018

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

154

View full text Add to dashboard Cite

Fig. 1: CalibNet estimates the extrinsic calibration parameters between a 3D LiDAR and a 2D camera. It takes as input an RGB image (a) from a calibrated camera, a raw LiDAR point cloud (b), and outputs a 6-DoF rigid-body transformation T that best aligns the two inputs. (c) shows the colorized point cloud output for a mis-calibrated setup, and (d) shows the output after calibration using our network. As shown, using the mis-calibrated point cloud to recover a colorized 3D map of the world results in an incoherent reconstruction. Notice how the 3D structures highlighted in (c) using red rectangles fail to project to their 2D counterparts. However, using the extrinsic calibration parameters predicted by CalibNet produces more consistent and accurate reconstructions (d), even for large initial mis-calibrations.Abstract-3D LiDARs and 2D cameras are increasingly being used alongside each other in sensor rigs for perception tasks. Before these sensors can be used to gather meaningful data, however, their extrinsics (and intrinsics) need to be accurately calibrated, as the performance of the sensor rig is extremely sensitive to these calibration parameters. A vast majority of existing calibration techniques require significant amounts of data and/or calibration targets and human effort, severely impacting their applicability in large-scale production systems. We address this gap with CalibNet: a geometrically supervised deep network capable of automatically estimating the 6-DoF rigid body transformation between a 3D LiDAR and a 2D camera in real-time. CalibNet alleviates the need for calibration targets, thereby resulting in significant savings in calibration efforts. During training, the network only takes as input a LiDAR point cloud, the corresponding monocular image, and the camera calibration matrix K. At train time, we do not impose direct supervision (i.e., we do not directly regress to the calibration parameters, for example). Instead, we train the network to predict calibration parameters that maximize the geometric and photometric consistency of the input images and point clouds. CalibNet learns to iteratively solve the underlying geometric problem and accurately predicts extrinsic calibration parameters for a wide range of mis-calibrations, without requiring retraining or domain adaptation. The project page is hosted at https://epiception.github.io/CalibNet

show abstract

RegNet: Multimodal sensor registration using deep neural networks

Cited by 173 publications

References 21 publications

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

CMRNet: Camera to LiDAR-Map Registration

CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks

Contact Info

Product

Resources

About