An Approach to Semantically Segmenting Building Components and Outdoor Scenes Based on Multichannel Aerial Imagery Datasets

Abstract

As-is building modeling plays an important role in energy audits and retrofits. However, in order to understand the source(s) of energy loss, researchers must know the semantic information of the buildings and outdoor scenes. Thermal information can potentially be used to distinguish objects that have similar surface colors but are composed of different materials. To utilize both the red–green–blue (RGB) color model and thermal information for the semantic segmentation of buildings and outdoor scenes, we deployed and adapted various pioneering deep convolutional neural network (DCNN) tools that combine RGB information with thermal information to improve the semantic and instance segmentation processes. When both types of information are available, the resulting DCNN models allow us to achieve better segmentation performance. By deploying three case studies, we experimented with our proposed DCNN framework, deploying datasets of building components and outdoor scenes, and testing the models to determine whether the segmentation performance had improved or not. In our observation, the fusion of RGB and thermal information can help the segmentation task in specific cases, but it might also make the neural networks hard to train or deteriorate their prediction performance in some cases. Additionally, different algorithms perform differently in semantic and instance segmentation.

Selecting Post-Processing Schemes for Accurate Detection of Small Objects in Low-Resolution Wide-Area Aerial Imagery

“…In addition, we need to improve the spatio-temporal processing scheme [14,20,81] by considering other hybrid approaches in the computer vision field, i.e., multi-scale structure information [6], multiple instance learning [100], and sparse representations [71,[101][102][103]. While employing any subset of methods from unsupervised, semi-supervised, and multi-task feature learning strategies [64,69,75,96,[104][105][106][107][108] on object detection, classification, and recognition, broader practical applications in the remote sensing domain may benefit from our research study.…”

Energy Efficiency Assessment for Buildings Based on the Generative Adversarial Network Structure

“…In building energy assessments, it is challenging to make accurate and reliable decisions on thermal estimates for specific structural areas affected by high energy losses. The research conducted in [16] presents an improved methodology for energy loss detection based on image segmentation and deep convolutional neural networks, which combines the information from thermal and visual images. In [17], the flight path for an unmanned aerial vehicle (UAV) for thermographic assessment was calculated based on the 3D model to achieve reproducible data collection.…”