Artificial Intelligence Assisted Infrastructure Assessment using Mixed Reality Systems

Abstract

Word count: 5,240 word texts + 3 tables x 250 words (each) = 5,990 words Submission Date: 08/01/2018 ABSTRACTConventional methods for visual assessment of civil infrastructures have certain limitations, such as subjectivity of the collected data, long inspection time, and high cost of labor. Although some new technologies (i.e. robotic techniques) that are currently in practice can collect objective, quantified data, the inspector's own expertise is still critical in many instances since these technologies are not designed to work interactively with human inspector. This study aims to create a smart, human-centered method that offers significant contributions to infrastructure inspection, maintenance, management practice, and safety for the bridge owners. By developing a smart Mixed Reality (MR) framework, which can be integrated into a wearable holographic headset device, a bridge inspector, for example, can automatically analyze a certain defect such as a crack that he or she sees on an element, display its dimension information in real-time along with the condition state. Such systems can potentially decrease the time and cost of infrastructure inspections by accelerating essential tasks of the inspector such as defect measurement, condition assessment and data processing to management systems. The human centered artificial intelligence (AI) will help the inspector collect more quantified and objective data while incorporating inspector's professional judgement. This study explains in detail the described system and related methodologies of implementing attention guided semi-supervised deep learning into mixed reality technology, which interacts with the human inspector during assessment. Thereby, the inspector and the AI will collaborate/communicate for improved visual inspection.

A review of computer vision–based structural health monitoring at local and global levels

“…In the application of CV-SHM, no matter LL or GL, the implementation of projective transform for camera calibration is always necessary. For CV-SHM-LL application, Karaaslan et al 10 implemented CV to estimate the camera pose of a headset and determine the length/width/area of detected cracks on structures. Then they assessed the structural condition as ''Good, Fair, Poor or Severe'' according to AASHTO codes.…”

“…Also, they applied the Class Active Map (CAM) with heat map extracted from the retrained neural network to visualize the possible spalling areas in the images. Karaaslan et al 10 first retrained the CNN architecture (SSD) for object detection purposes by using transfer learning to detect spalling in an image, and then proposed an attention guided segmentation network (SegNet) to segment spalling from concrete columns and walls. The guided segmentation with human aided operation does not need full image search and can improve the accuracy.…”

“…CNN for image classification (new developed): Cha et al, 33 Yokoyama and Matsumoto, 34 Kim and Cho, 37 Chen and Jahanshahi 38 CNN (transfer learning: VGG-16): da Silva and de Lucena 35 CNN (transfer learning: AlexNet): Dorafshan et al 36 CNN (transfer learning: GoogLeNet): Jang et al 39 CNN for object detection (SSD): Karaaslan et al 10…”

“…CNN (transfer learning: VGG-16): Gao and Mosalam 69 CNN (transfer learning: SSD): Karaaslan et al 10 CNN (transfer learning: SegNet): Karaaslan et al 10…”

“…The sizes of the detected regions can be different. Karaaslan et al 10 retrained the VGG-16 network weights in Single Shot MultiBox Detector (SSD) architecture to recognize crack regions with a bounding box and applied the SegNet model to segment the cracks inside the bounding box.…”