Deep learning‐based object identification with instance segmentation and pseudo‐LiDAR point cloud for work zone safety

Abstract

Automated object identification in three‐dimensional (3D) space is crucial for work zone safety, such as compliance with construction rules and preventing workplace injuries and deaths. However, it is greatly challenged by some factors like high‐quality detection, high‐quality instance segmentation, few engineering object datasets with masks, and accurate 3D object understanding due to scale variations and limited cues in the 3D world. Traditional hand‐crafted methods suffer from these challenges. Our key insight is to use 2D object detection, instance segmentation and camera vision to compute pseudo‐light detection and ranging (LiDAR) point cloud for 3D object identification. On the one hand, an enhanced feature pyramid network is proposed to extract more fine‐grained object features, and an improved cascade mask R‐CNN is applied to detect bounding boxes and masks for all 2D objects efficiently. Moreover, the AIM dataset for heavy equipment detection is augmented, and a new object class with the bounding box and mask is added. On the other hand, pseudo‐LiDAR point clouds of objects based on bounding boxes and masks are recovered from a monocular image by deep learning, automatic camera parameter estimation, vision‐based method, and space filter. Extensive experiments and analyses show that the new methodology can identify 3D objects and automatically analyze work zone safety. The proposed object detection model has achieved state‐of‐the‐art results on the AIM dataset and 97.2% in mean average precision for the augmented dataset. The collision detection model using pseudo‐LiDAR point cloud has obtained 95.99% in accuracy. The new model will serve as a baseline to support 3D object identification research for other 3D tasks.

Severe rail wear detection with rail running band images

“…Wang et al (2019) employed pseudo-LiDAR (Laser Radar) technology for 3D target detection from 2D images, in which the depth map generated from an image is input to detectors. Shen et al (2021) used an estimated depth map to create pseudo-LiDAR point clouds of objects for work zone safety. In addition, benefiting from the development of unsupervised and semisupervised-based depth estimation models, many studies have demonstrated that depth and RGB fusion for classification improve the performance without requiring ground-truth training data (Ding et al, 2020;Ouyang et al, 2020;Yoo et al, 2020).…”

An integrated low-cost system for object detection in underwater environments

“…Other significative, but less related, works are devoted to visually recognising objects either for zone safety purposes ( [34]), or for structural health assessments of infrastructures ( [25]). They resort to sophisticated techniques which go beyond to the use of CNNs and they achieve precision scores of 97.2% and accuracy scores of 95.99%.…”

A self‐supervised monocular depth estimation model with scale recovery and transfer learning for construction scene analysis

“…The above models for depth information are not tested and verified on a publicly available dataset, so it is difficult to evaluate these traditional learning paradigms. Additionally, the pseudo‐light detection and ranging (LiDAR) point cloud method is effective for 3D spatial relationships recognition (Liu et al., 2021; Qian et al., 2020), and it is also used for work zone safety (Shen et al., 2021), which reconstructs pseudo‐LiDAR point cloud by depth prediction. However, the method suffers from scalar recovery and representation transferability when estimating accurate depth information.…”

“…However, collecting training data for various scenes and tasks is a formidable challenge (Godard et al., 2019). Currently, there are some datasets with ground‐truth depth information, such as the KITTI dataset (Geiger et al., 2013), mainly used for autonomous driving applications, but it is extremely difficult to obtain such publicly available datasets in the construction industry (Shen et al., 2021; Yan et al., 2020). Therefore, the self‐supervised learning method is an alternative, and it makes use of extensive unlabeled data to train deep representations and approaches and even outperforms the supervised learning method (Ericsson et al., 2021).…”

“…Due to the difference in feature space and data distribution between autonomous driving and construction scene analysis, the knowledge transfer between them would improve representation learning in a downstream task. However, although we can solve this transfer learning problem between autonomous driving tasks with ease, such as the model transfer from the DDAD dataset to the KITTI‐3D dataset (Park et al., 2021), it is challenging for task transfer to a construction scene (Shen et al., 2021), where no depth ground truth like LiDAR data is available. Therefore, when depth ground truth is unavailable in a construction scene, transferring and evaluating a model is a challenging problem.…”