MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure

Hoskere, Vedhus; Narazaki, Yasutaka; Hoang, Thu Trang; Spencer, Billie F.

doi:10.1007/s13349-020-00409-0

Cited by 88 publications

(40 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Such algorithms use image processing pipelines to identify damage such as concrete cracking, spalling, and exposed rebar [ 10 , 11 , 12 , 13 , 14 ]. More recently, classifiers using convolutional neural networks [ 15 , 16 , 17 , 18 , 19 , 20 ] have successfully been applied to automatically identify structural component type and structural damage. Beyond identifying structural damage, Paal et al [ 14 ] developed a classifier to automatically estimate maximum column drift demand experienced during an earthquake from an image of a damaged column.…”

Section: Introductionmentioning

confidence: 99%

Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework

Levine

Spencer

2022

Sensors

View full text Add to dashboard Cite

Computer vision has shown potential for assisting post-earthquake inspection of buildings through automatic damage detection in images. However, assessing the safety of an earthquake-damaged building requires considering this damage in the context of its global impact on the structural system. Thus, an inspection must consider the expected damage progression of the associated component and the component’s contribution to structural system performance. To address this issue, a digital twin framework is proposed for post-earthquake building evaluation that integrates unmanned aerial vehicle (UAV) imagery, component identification, and damage evaluation using a Building Information Model (BIM) as a reference platform. The BIM guides selection of optimal sets of images for each building component. Then, if damage is identified, each image pixel is assigned to a specific BIM component, using a GrabCut-based segmentation method. In addition, 3D point cloud change detection is employed to identify nonstructural damage and associate that damage with specific BIM components. Two example applications are presented. The first develops a digital twin for an existing reinforced concrete moment frame building and demonstrates BIM-guided image selection and component identification. The second uses a synthetic graphics environment to demonstrate 3D point cloud change detection for identifying damaged nonstructural masonry walls. In both examples, observed damage is tied to BIM components, enabling damage to be considered in the context of each component’s known design and expected earthquake performance. The goal of this framework is to combine component-wise damage estimates with a pre-earthquake structural analysis of the building to predict a building’s post-earthquake safety based on an external UAV survey.

show abstract

Section: Introductionmentioning

confidence: 99%

Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework

Levine

Spencer

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Researchers have also sought to incorporate context of the damage and information from the entire structure to contribute to a structural assessment using deep learning methods. For example, Hoskere, et al [14,15] proposed the use of deep-learning based semantic segmentation for multiple types of damage and materials. The proposed methodology was extended to the semantic segmentation of scenes, components, and damage in reinforced concrete buildings in [16].…”

Section: Introductionmentioning

confidence: 99%

Physics-Based Graphics Models in 3D Synthetic Environments Enabling Autonomous Vision-Based Structural Inspections

Hoskere¹,

Narazaki²,

Spencer³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Manual visual inspections typically conducted after an earthquake are high-risk, subjective, and time-consuming. Delays from inspections often exacerbate the social and economic impact of the disaster on affected communities. Rapid and autonomous inspection using images acquired from unmanned aerial vehicles offer the potential to reduce such delays. Indeed, a vast amount of re-search has been conducted toward developing automated vision-based methods to assess the health of infrastructure at the component and structure level. Most proposed methods typically rely on images of the damaged structure, but seldom consider how the images were acquired. To achieve autonomous inspections, methods must be evaluated in a comprehensive end-to-end manner, incorporating both data acquisition and data processing. In this paper, we leverage recent advances in computer generated imagery (CGI) to construct a 3D synthetic environment for simulation of post-earthquake inspections that allows for comprehensive evaluation and valida-tion of autonomous inspection strategies. A critical issue is how to simulate and subsequently render the damage in the structure after an earthquake. To this end, a high-fidelity nonlinear finite element model is incorporated in the synthetic environment to provide a representation of earthquake-induced damage; this finite element model, combined with photo-realistic rendering of the damage, is termed herein a physics-based graphics models (PBGM). The 3D synthetic en-vironment with PBGMs provide a comprehensive end-to-end approach for development and validation of autonomous post-earthquake strategies using UAVs, including: (i) simulation of path planning of virtual UAVs and image capture under different environmental conditions; (ii) au-tomatic labeling of captured images, potentially providing an infinite amount of data for training deep neural networks; (iii) availability of the ground truth damage state from the results of the finite-element simulation; and (iv) direct comparison of different approaches to autonomous as-sessments. Moreover, the synthetic data generated has the potential to be used to augment field datasets. To demonstrate the efficacy of PBGMs, models of reinforced concrete moment-frame buildings with masonry infill walls are examined. The 3D synthetic environment employing PBGMs is shown to provide an effective testbed for development and validation of autonomous vision-based post-earthquake inspections that can serve as an important building block for ad-vancing autonomous data to decision frameworks.

show abstract

“…Health monitoring is an essential process in ensuring the safety and serviceability of civil infrastructure like bridges and container cranes (Rao et al, 2020;Saleem et al, 2020;Stein, 2018). Current practice for assessing structural health of container cranes is mainly based on visual inspections by human operators (Hoskere et al, 2020). Container bridges or ship-toshore cranes are the common means in seaport container terminals for loading and unloading the containers from container ships.…”

Section: Introductionmentioning

confidence: 99%

Drone-Based Container Crane Inspection: Concept, Challenges and Preliminary Results

Maboudi

Alamouri

López

et al. 2021

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

Abstract. Container crane inspection is a very important task to maintain their uninterrupted operation. Nevertheless, this is a costly and time-consuming activity if performed manually. Recently, image-based detection of surface damages or changes using drones has gained increasing interest in industry; especially when objects of interest have a complex structure like container cranes. One main aim of this paper is a single-epoch image analysis which will also serve later for multi-epoch processing. It provides reliable information about current defects that may lead to big damages if not inspected by experts. Naïve Bayes classifier is employed to classify the images in different classes of which critical defects and especially rust is important. The preliminary results show that the precision on the target class reached about 99%. However, 87% percent recall in this class is not enough and it should be improved for this application.Having a large dataset requires an efficient data management system to provide users and decision makers with the information needed. In addition, in order to foster full automation, the aforementioned image analysis component should have a direct connection to the database and thus is able to query image and semantic information. We therefore introduce the second aim of our research, that is a concept for database design. Here, not only the raw data and the final results are integrated but also the intermediate results. At the same time, the database concept is connected to an integrated client interface that allows retrieving data of interest in a virtual globe.

show abstract

MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure

Cited by 88 publications

References 45 publications

Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework

Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework

Physics-Based Graphics Models in 3D Synthetic Environments Enabling Autonomous Vision-Based Structural Inspections

Drone-Based Container Crane Inspection: Concept, Challenges and Preliminary Results

Contact Info

Product

Resources

About