Automatic Object Removal With Obstructed Façades Completion Using Semantic Segmentation and Generative Adversarial Inpainting

Abstract

Automatic object removal with obstructed façades completion in the urban environment is essential for many applications such as scene restoration, environmental impact assessment, and urban mapping. However, the previous object removal typically requires a user to manually create a mask around unwanted objects and obtain background façade information in advance, which would be labor-intensive when implementing multitasking projects. Moreover, accurately detecting objects to be removed in the cityscape and inpainting the static obstructed building façade to obtain plausible images are the main challenges for this objective. To overcome these difficulties, this study addresses the object removal with the façade inpainting problem from the following two aspects. First, we proposed an image-based cityscape elimination method for automatic object removal and façade inpainting by applying semantic segmentation to detect several classes, including pedestrians, riders, vegetation, and cars, as well as using generative adversarial networks (GANs) for filling detected regions by background textures and patching information from street-level imagery. Second, we proposed a workflow to filter unoccluded building façades from street view images automatically and tailored a dataset for the GAN-based image inpainting model with original and mask images. Furthermore, several full-reference image quality assessment (IQA) metrics are introduced to evaluate the generated image quality. Validation results demonstrated the feasibility and effectiveness of our proposed method, and the synthetic image is visually realistic and semantically consistent.INDEX TERMS Generative adversarial networks, semantic segmentation, automatic object removal, façade inpainting, street view images.

Uncovering Bias in Objective Mapping and Subjective Perception of Urban Building Functionality: A Machine Learning Approach to Urban Spatial Perception

“…Machine learning is extensively applied in urban planning studies, with algorithms such as Random Forest (RF) [37], Convolutional Neural Networks (CNN) [7], and deep learning methodologies proving ideal for classification and pattern analysis of geo-observed data. Generative Adversarial Networks (GAN) have been utilized for simulating urban patterns [38,39]. Contributions from deep learning (DL) and machine learning (ML) methods to the evolution of models in various aspects of prediction, planning, and uncertainty analysis in smart cities and urban development have been notable [40,41], providing support for urban planning and decision making [42].…”

Pareto Optimized Large Mask Approach for Efficient and Background Humanoid Shape Removal

“…The missing area can then be synthesized using large-scale boundconstrained optimization based on the CNN encoding properties of similar nearby regions [75]. Using such semantic segmentation to recognize different groups of objects, Paper [76] developed an image-based object elimination approach for automated object removal and inpainting with generative adversarial networks, automating the city-scape object detection and processing, while paper [77] applied a similar way to indoor-scape scenarios and paper [78] for autonomous driving data preparation. Li et al [79] proposed a more universal approach, using three trainable GAN modules, including flow completion, feature propagation, and content hallucination, which gave improved outcomes in object removal accuracy.…”

A Virtual Reality-Based Tool with Human Behavior Measurement and Analysis for Feedback Design of the Indoor Light Environment

“…Previous studies on human behavior analysis used multi-agent-based behavioral simulation and field observation using manual labor or monitoring techniques (e.g., camera, Wi-Fi, UWB, Bluetooth) [10]. They either fail to simulate realistic complex and finegrained behavior or require strict field experiments, rely on a physically built environment, and are challenging to use in the schematic design phase.…”