Machine-learned 3D Building Vectorization from Satellite Imagery

Abstract

We propose a machine learning based approach for automatic 3D building reconstruction and vectorization. Taking a single-channel photogrammetric digital surface model (DSM) and a panchromatic (PAN) image as input, we first filter out non-building objects and refine the building shapes of the input DSM with a conditional generative adversarial network (cGAN). The refined DSM and the input PAN image are then used through a semantic segmentation network to detect edges and corners of building roofs. Later, a set of vectorization algorithms are proposed to build roof polygons. Finally, the height information from refined DSM is processed and added to the polygons to obtain a fully vectorized level of detail (LoD)-2 building model. We verify the effectiveness of our method on large-scale satellite images, where we obtain state-of-the-art performance.

Procedural Roof Generation From a Single Satellite Image

“…However, they require high resolution aerial images and annotate the dataset for roof ridges and building boundaries. In [MPBF20; WZB21], they present deep learning based approaches for automatic 3D building reconstruction. However, they need elevation data (e.g., DSM) for training and their results are not regularized.…”

ResDepth: A Deep Residual Prior For 3D Reconstruction From High-resolution Satellite Images

“…More recently, [23], [55] incorporate a multi-task loss to jointly optimize for building shapes, roof orientations, and roof type classification. The latest extensions introduce skip connections in the generator [22], learned weights of the multitask loss terms [22], and an additional attention module [19]. In comparison, our preliminary work [25] uses a much simpler U-Net architecture [24] trained with stereo guidance and a single 1 -loss on pixel-wise differences to reference heights.…”

“…Furthermore, they quickly reach their limits when the initial surface estimate is corrupted by significant noise, such that the correct primitive cannot be recognized. In summary, we posit that, to further improve DSMs and to facilitate downstream modeling applications, such as the automatic vectorization of 3D buildings [19], one must go beyond the simplistic smoothness assumption of conventional stereo reconstruction and inject more prior knowledge about plausible surface shapes; while at the same time maintaining enough flexibility to accommodate the variety of real surfaces beyond schematic "box+roof" primitives.…”

“…In the domain of satellite stereo, [20] were perhaps the first who leveraged a learning-based approach to see image pixels in context and convert raw, noisy height measurements into a more accurate DSM. Later extensions introduced a single ortho-rectified panchromatic satellite view to guide the DSM refinement [21], an improved network architecture [19], [22], and a multi-task loss [23] to jointly optimize for building shapes, roof orientations, and roof type classification. All these methods seem to imply that a rather involved network architecture (cGAN) and multiple loss terms, including adversarial losses, are needed to refine an imperfect DSM into a more accurate and detailed one.…”

From Raster Predictions to Vector Layers of Buildings: A Computational Geometry Approach

“…PolyMapper [8] alleviates this problem by using recurrent neural networks to produce vectorial city maps, but polygons still offer no geometric guarantees such as orthogonality, or absence of overlap. This issue is partially addressed in [9], where an architecture similar to [8] is used to favor the extraction of regular polygons from remote data, but this method also requires a DSM. Zorzi et al [10], for their part, look into the problem of obtaining regular rooftops by regularizing the raster contours of the prediction through generative adversarial networks.…”