Inspection-Nerf: Rendering Multi-Type Local Images for Dam Surface Inspection Task Using Climbing Robot and Neural Radiance Field

Abstract

For the surface defects inspection task, operators need to check the defect in local detail images by specifying the location, which only the global 3D model reconstruction can’t satisfy. We explore how to address multi-type (original image, semantic image, and depth image) local detail image synthesis and environment data storage by introducing the advanced neural radiance field (Nerf) method. We use a wall-climbing robot to collect surface RGB-D images, generate the 3D global model and its bounding box, and make the bounding box correspond to the Nerf implicit bound. After this, we proposed the Inspection-Nerf model to make Nerf more suitable for our near view and big surface scene. Our model use hash to encode 3D position and two separate branches to render semantic and color images. And combine the two branches’ sigma values as density to render depth images. Experiments show that our model can render high-quality multi-type images at testing viewpoints. The average peak signal-to-noise ratio (PSNR) equals 33.99, and the average depth error in a limited range (2.5 m) equals 0.027 m. Only labeled 2% images of 2568 collected images, our model can generate semantic masks for all images with 0.957 average recall. It can also compensate for the difficulty of manual labeling through multi-frame fusion. Our model size is 388 MB and can synthesize original and depth images of trajectory viewpoints within about 200 m2 dam surface range and extra defect semantic masks.

Façade Protrusion Recognition and Operation-Effect Inspection Methods Based on Binocular Vision for Wall-Climbing Robots

“…Developments in convolutional neural networks and image-processing technologies have driven the automated application of wall-climbing robots, making machine vision increasingly important [4,5]. The application of machine vision not only enables efficient and highly accurate product manufacturing in factory production but also allows robots to replace humans in hazardous environments for inspection and operational tasks.…”