Augmented Reality and Deep Learning towards the Management of Secondary Building Assets

Abstract

The retrieval of as-is information for existing buildings is a prerequisite for effectively operating facilities, through the creation or updating of Building/Asset Information Models (BIM/AIM), or Digital Twins. At present, many studies focus on the capture of geometry for the modelling of primary components, overlooking the fact that many recurring actions need to be conducted on assets inside buildings. Furthermore, highly accurate survey techniques like laser scanning need long offsite processing for object recognition. Performing such process on site would dramatically impact efficiency and also prevent the need to revisit the site in the case of insufficient/incomplete data. In this paper, an Augmented Reality (AR) system is proposed enabling inventory, information retrieval and information update directly on-site. It would reduce post-processing work and avoid loss of information and unreliability of data. The system has a Head-Mounted Display (HMD) AR interface that lets the technician interact handsfree with the real world and digital information contained in the BIM/AIM. A trained Deep Learning Neural Network operates the automatic recognition of objects in the field of view of the user and their placement into the digital BIM. In this paper, two uses cases are described: one is the inventory of small assets inside buildings to populate a BIM/AIM, and the second is the retrieval of relevant information from the AIM to support maintenance operations. Partial development and feasibility tests of the first use case applied to fire extinguishers, have been carried out to assess the feasibility and value of this system.

BIM-Based Digital Twin and XR Devices to Improve Maintenance Procedures in Smart Buildings: A Literature Review

“…This type is mainly used to provide an immersive environment for its user, leaving his or her hands free to interact with the real environment. There are two main subtypes of HMD that can be found in the literature: the optical and/or video see-through HMDs, which let the user see the real-word with immersive augmentations [25,26,43,47,66,67,[70][71][72][73]75,78,80] (see Figure 6b), and the occluded HMDs, where the user is entirely immersed in a virtual environment [25,47,75,78,79] (see Figure 6c). While the occluded HMDs are mainly seen as VR devices, the see-through HMDs are mostly used for AR and MR applications.…”

“…Prouzeau et al presented another example of the use of this interaction method by allowing the creation of specific zones in which the user must stand in order to view the information displayed to him [25] (see Figure 7). Some authors have proposed using another interaction method to also estimate the orientation of the user, namely gaze tracking, which can be coupled with head tracking [25,66,71,72]. One of the main uses that can be made of gaze/head tracking is to select the augmentations that will be displayed, such as the time-series specific to a piece of equipment and which is displayed only when the user aims at it, as shown by Prouzeau et al [25].…”

“…Through their different interaction methods, several complex processes can be reached intuitively. Some authors have decided to use the device's camera feed to collect various data using object recognition algorithms (e.g., segmentation algorithms applied to 3D point clouds [81], comparison with grid data from BIM [71]). Once an object has been identified, the information can be used for various purposes, such as to identify the discrepancies between the digital and real environment [71] (see Figure 8) or to link specific inspection feedback to the equipment's digital model [66].…”

“…to 3D point clouds [81], comparison with grid data from BIM [71]). Once an object has been identified, the information can be used for various purposes, such as to identify the discrepancies between the digital and real environment [71] (see Figure 8) or to link specific inspection feedback to the equipment's digital model [66].…”

“…to 3D point clouds [81], comparison with grid data from BIM [71]). Once an object has been identified, the information can be used for various purposes, such as to identify the discrepancies between the digital and real environment [71] (see Figure 8) or to link specific inspection feedback to the equipment's digital model [66]. Even though VR devices and common digital environments (CDE) are often used for evaluations of skills and training purposes, some authors have tried to show their benefits for collaboration [65,78].…”