Facade Reconstruction for Textured Lod2 Citygml Models Based on Deep Learning and Mixed Integer Linear Programming

Abstract

<p><strong>Abstract.</strong> The paper describes a workflow for generating LoD3 CityGML models (i.e. semantic building models with structured facades) based on textured LoD2 CityGML models by adding window and door objects. For each wall texture, bounding boxes of windows and doors are detected using “Faster R-CNN”, a deep neural network. We evaluate results for textures with different resolutions on the ICG Graz50 facade dataset. In general, detected bounding boxes match very well with the rectangular shape of most wall openings. Thus, no further classification of shapes is required. Windows are typically aligned to rows and columns, and only a few different types of windows exist for each facade. However, the neural network proposes rectangles of varying sizes, which are not always aligned perfectly. Thus, we use post-processing to obtain a more realistic appearance of facades. Window and door rectangles get aligned by solving a mixed integer linear optimization problem, which automatically leads to a clustering of these openings into few different classes of window and door types. Furthermore, an a-priori knowledge about the number of clusters is not required.</p>

Fast and Regularized Reconstruction of Building Façades From Street-View Images Using Binary Integer Programming

“…It has been used for arrangements of the 2D boundaries and 3D planes (Monszpart et al, 2015), reconstruction of surface meshes (Boulch et al, 2014, Nan, Wonka, 2017, modeling of the roof structures of the LOD-2 models (Goebbels, Pohle-Fröhlich, 2019) and the façades (Hensel et al, 2019). However, most of them formulated the optimization problem as MILP (Goebbels, Pohle-Fröhlich, 2019, Hensel et al, 2019 or even mixed integer non-linear programming (Monszpart et al, 2015), which has unknowns in both spaces of integer and real values. Unfortunately, this kind of problems raised up in the operational research has no efficient solvers for large scale problems, even using state-of-the-art commercial libraries (Gurobi, 2014).…”

“…After initial extraction of the bounding boxes of the building façade, we then use BIP to restore the spatial regularity of the windows, doors and balconies, inspired by previous work (Hensel et al, 2019). Although the MILP method has been successfully used in many studies (Boulch et al, 2014, Hensel et al, 2019, in our pipeline, because we are aiming at an interactive reconstruction pipeline, the runtime should be kept reason-ably low. In the following, we describe our reformulated problem setup using BIP instead of MILP.…”

“…After extracting the initial primitives, we have N bounding boxes for each image, and each bounding box is uniquely determined by a set of four parameters (x, y, w, h) , where (x, y) and (w, h) are coordinate of the upper left corner and size of the bounding box, respectively (Figure 1a). Instead of directly optimizing these parameters that are real values using MILP (Hensel et al, 2019), we cast it into a model selection problem using BIP.…”

“…Semantic objects such as windows, doors, and balconies are important components of a building façade. Extracting them (Hoegner, Stilla, 2015) and arranging them in a regularized manner (Hensel et al, 2019) are two important steps towards structured LOD-3 reconstruction (Zhu et al, 2020). And the street-view image is arguably the best option for the above objectives due to the public availability and effectiveness in collecting, such as the Google street map (Anguelov et al, 2010).…”

“…Recently, a general and promising approach to align different objects of building façades using Mixed Integer Linear Programming (MILP) was proposed (Hensel et al, 2019). However, in our practice the MILP is too complex to solve, which requires prohibitively high runtime consumption.…”