Dense Omnidirectional RGB‐D Mapping of Large‐scale Outdoor Environments for Real‐time Localization and Autonomous Navigation

Abstract

This paper presents a novel method and innovative apparatus for building three‐dimensional (3D) dense visual maps of large‐scale unstructured environments for autonomous navigation and real‐time localization. The main contribution of the paper is focused on proposing an efficient and accurate 3D world representation that allows us to extend the boundaries of state‐of‐the‐art dense visual mapping to large scales. This is achieved via an omnidirectional key‐frame representation of the environment, which is able to synthesize photorealistic views of captured environments at arbitrary locations. Locally, the representation is image‐based (egocentric) and is composed of accurate augmented spherical panoramas combining photometric information (RGB), depth information (D), and saliency for all viewing directions at a particular point in space (i.e., a point in the light field). The spheres are related by a graph of six degree of freedom (DOF) poses (3 DOF translation and 3 DOF rotation) that are estimated through multiview spherical registration. It is shown that this world representation can be used to perform robust real‐time localization (in 6 DOF) of any configuration of visual sensors within their environment, whether they be monocular, stereo, or multiview. Contrary to feature‐based approaches, an efficient direct image registration technique is formulated. This approach directly exploits the advantages of the spherical representation by minimizing a photometric error between a current image and a reference sphere. Two novel multicamera acquisition systems have been developed and calibrated to acquire this information, and this paper reports for the first time the second system. Given the robustness and efficiency of this representation, field experiments demonstrating autonomous navigation and large‐scale mapping will be reported in detail for challenging unstructured environments, containing vegetation, pedestrians, varying illumination conditions, trams, and dense traffic.

Cloud-Based Geospatial 3D Image Spaces—A Powerful Urban Model for the Smart City

“…Their published approach, however, sacrifices some of the accuracy by projecting and mosaicking the original multi-head panoramic imagery and depth information onto a cylindrical panorama [28]. A recently published and similar approach uses collections of spherical RGB-D panoramas of large-scale urban environments for autonomous navigation and real-time localization [14]. The spherical RGB-D panoramas, referred to as "augmented spherical key-frames", are also generated by warping and mosaicking multiple images onto a sphere.…”

“…Image data for both models are acquired using multi-head camera systems. By considering all sensors of a panoramic head to have a unique center of projection [14], both approaches reduce the relative and absolute 3D measurement accuracy from a potential cm-level to the dm-level or even lower. Other approaches such as the one described for the Stereopolis II system [7], preserve the original perspective image geometry and are thus not affected by the accuracy degradation.…”

“…While there is no widely accepted taxonomy of (3D) urban models, there are a number of helpful classifications. Meilland et al [14], for example, distinguish between 3D parametric models and image-based key-frame models, but they do not cover point cloud-based models, which play an important role in urban modeling. In an earlier overview of urban models [15], the authors distinguish between geometric 3D models, image-based models, and a rich point cloud model.…”

“…These include: monoscopic, stereoscopic [7], panoramic [23] or RGB-D imagery [7,8,14], 3D point clouds [15] or combinations thereof [24]. Native urban models do not aim at complete 3D reconstructions of entire objects or urban scenes.…”

“…This temporal coherence is particularly important in urban environments with numerous moving objects such as cars, trams or pedestrians (see Section 4.4). In outdoor urban environments, the requirement of dense and temporally-coherent RGB-D values with a sufficiently high spatial resolution can currently only be met by stereo or trinocular camera setups [6,7,14,20] and a subsequent depth extraction using dense image matching (see Section 6). Once active range imaging sensors will reliably work in bright daylight and provide a sufficiently high spatial resolution, they might become the sensor technology of choice for acquiring urban 3D image spaces.…”