Multimodal image inpainting for an autonomous robot navigation application

Gapon, Nikolay; Воронин, В. В.; Semenishchev, Evgeny A.; Ilyukhin, Yurii; Bezuglov, D. A.; Zelenskii, A. A.

doi:10.1117/12.2594476

Cited by 11 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…belonging to this pixel is selected [9]. Next, there are blocks , in available pixels for which the Euclidean metric over the spectrum is minimal [10]. Pixel values in area adjacent to the pixel with maximum priority are recovered by the corresponding pixels from the found areas in the area of available pixels using a fusion neural network.…”

Section: Reconstruction Methodsmentioning

confidence: 99%

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Gapon

Voronin

Sizyakin

et al. 2023

Dimensional Optical Metrology and Inspection for Practical Applications XII

View full text Add to dashboard Cite

A system for determination the distance from the robot to the scene is useful for object tracking, and 3-D reconstruction may be desired for many manufacturing and robotic tasks. While the robot is processing materials, such as welding parts, milling, drilling, fragments of materials fall on the camera installed on the robot, introducing unnecessary information when building a depth map, as well as the emergence of new lost areas, which leads to incorrect determination of the size of objects. There is a problem comprising a decrease in the accuracy of planning the movement trajectory caused by incorrect sections on the depth map because of incorrect distance determination of objects. We present a two-stage approach combining defect detection and depth reconstruction algorithms. The first step is image defects detection based on convolutional auto-encoder (U-Net) and deep feature fusion network (DFFN-Net). The second step is a depth map reconstruction with the exemplar-based and the anisotropic gradient concepts. The proposed modified block fusion algorithm uses a local image descriptor obtained by an automatic encoder for image reconstruction, which extracts image features and depth maps using a decoding network. Our technique outperforms the state-of-the-art methods quantitatively in reconstruction accuracy on RGB-D benchmark for evaluating manufacturing vision systems.

show abstract

Section: Reconstruction Methodsmentioning

confidence: 99%

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Gapon

Voronin

Sizyakin

et al. 2023

Dimensional Optical Metrology and Inspection for Practical Applications XII

View full text Add to dashboard Cite

show abstract

“…belonging to this pixel is selected (Figure 10) [14][15]. Using allows you to correctly take into account the shape of the restoration area and not capture unnecessary boundaries [3,10].…”

Section: Inpaintingmentioning

confidence: 99%

Deep learning-based depth map defect removal for industrial applications

Voronin,

Gapon,

Zhdanova

et al. 2023

Optical Metrology and Inspection for Industrial Applications X

View full text Add to dashboard Cite

A system for determining the distance from the robot to the scene is useful for object tracking, and 3-D reconstructions may be desired for many manufacturing and robotic tasks. While the robot is processing materials, such as welding parts, milling, drilling, etc., fragments of materials fall on the camera installed on the robot, introducing unnecessary information when building a depth map, as well as the emergence of new lost areas, which leads to incorrect determination of the size of objects. There is a problem comprising a decrease in the accuracy of planning the movement trajectory caused by wrong sections on the depth map because of erroneous distance determination to objects. We present an approach combining defect detection and depth reconstruction algorithms. The first step for image defect detection is based on a convolutional auto-encoder (U-Net). The second step is a depth map reconstruction using a spatial reconstruction based on a geometric model with contour and texture analysis. We apply contour restoration and texture synthesis for image reconstruction. A method is proposed for restoring the boundaries of objects in an image based on constructing a composite curve by cubic splines. Our technique outperforms the state-of-the-art methods quantitatively in reconstruction accuracy on the RGB-D benchmark for evaluating manufacturing vision systems.

show abstract

“…Using allows you to correctly take into account the shape of the restoration area and not capture extra boundaries that can lead to incorrect image reconstruction. Next, there are blocks , in the area of available pixels for which the Euclidean metric over the spectrum is minimal [18]:…”

Section: Proposed Inpainting Algorithmmentioning

confidence: 99%

A vision system using depth inpainting for virtual content reconstruction in Augmented Reality

Voronin

Gapon

Khamidullin

et al. 2022

Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies VI

View full text Add to dashboard Cite

Augmented Reality (AR) applications demand realistic rendering of virtual content in a variety of environments, so they require an accurate description of the 3-D scene. In most case AR system is equipped with Time-of-Flight (ToF) cameras to provide real-time scene depth maps, but they have problems that affect the quality of depth data, which ultimately makes them difficult to use for AR. Such defects appear because of poor lighting, specular or fine-grained surfaces of objects. As a result, the effect of increasing the boundaries of objects appears, and the overlapping of objects makes it impossible to distinguish one object from another. The article presents an approach based on a modified algorithm for searching for similar blocks using the concept of anisotropic gradient. A proposed modified exemplar block-based algorithm uses the autoencoder-learned local image descriptor for image inpainting, that extract the features of images, and the depth image by a decoding network. The encoder consists of a convolutional layer and a dense block, which also consists of convolutional layers. We also show the application for the proposed vision system using depth inpainting for virtual content reconstruction in augmented reality. Analysis of the results of the study shows that the proposed method allows you to correctly restore the boundaries of objects on the image of the depth map. Our system quantitatively outperforms state-of-the-art methods in terms of reconstruction accuracy in the real and simulated benchmark datasets.

show abstract

Multimodal image inpainting for an autonomous robot navigation application

Cited by 11 publications

References 15 publications

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Defect detection and removal for depth map quality enhancement in manufacturing with deep learning

Deep learning-based depth map defect removal for industrial applications

A vision system using depth inpainting for virtual content reconstruction in Augmented Reality

Contact Info

Product

Resources

About