Restoration of Lighting Parameters in Mixed Reality Systems Using Convolutional Neural Network Technology Based on RGBD Images

Sorokin, Maxim; Zhdanov, Dmitry; Zhdanov, Andrey; Potemin, Igor S.; Bogdanov, N. N.

doi:10.1134/s0361768820030093

Cited by 6 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This is often not so [ 14 ]. Lighting prediction methods [ 15 , 16 , 17 , 18 ] can help here. They have become a quite common task for mixed reality systems.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Image Synthesis Pipeline for CNN-Based Sensing Systems

Frolov

Faizov

Shakhuro

et al. 2022

Sensors

View full text Add to dashboard Cite

The rapid development of machine learning technologies in recent years has led to the emergence of CNN-based sensors or ML-enabled smart sensor systems, which are intensively used in medical analytics, unmanned driving of cars, Earth sensing, etc. In practice, the accuracy of CNN-based sensors is highly dependent on the quality of the training datasets. The preparation of such datasets faces two fundamental challenges: data quantity and data quality. In this paper, we propose an approach aimed to solve both of these problems and investigate its efficiency. Our solution improves training datasets and validates it in several different applications: object classification and detection, depth buffer reconstruction, panoptic segmentation. We present a pipeline for image dataset augmentation by synthesis with computer graphics and generative neural networks approaches. Our solution is well-controlled and allows us to generate datasets in a reproducible manner with the desired distribution of features which is essential to conduct specific experiments in computer vision. We developed a content creation pipeline targeted to create realistic image sequences with highly variable content. Our technique allows rendering of a single 3D object or 3D scene in a variety of ways, including changing of geometry, materials and lighting. By using synthetic data in training, we have improved the accuracy of CNN-based sensors compared to using only real-life data.

show abstract

“…This is often not so [ 14 ]. Lighting prediction methods [ 15 , 16 , 17 , 18 ] can help here. They have become a quite common task for mixed reality systems.…”

Section: Related Workmentioning

confidence: 99%

“…This is because we do not really know the lighting settings for the augmented image. As we already mentioned, the lighting prediction methods [ 15 , 16 , 17 , 18 ] can help here. However, their application has a limitation: different methods should be used for different datasets (for example, indoor and outdoor scenes).…”

Section: Experimental Evaluationmentioning

confidence: 99%

Image Synthesis Pipeline for CNN-Based Sensing Systems

Frolov

Faizov

Shakhuro

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Так, например, существуют различные работы по восстановлению положения источников света через обучение нейронных сетей [19], [20]. В этом случае происходит сегментация всей сцены с целью работы с координатами относительно заданной измерительной системы.…”

Section: обзор методов решения проблем смешанной реальностиunclassified

A method for evaluating the correctness of the illumination of virtual objects in mixed reality systems

Antonov

Sorokin

Zhdanov

2021

Proceedings of the 31th International Conference on Computer Graphics and Vision. Volume 1

View full text Add to dashboard Cite

The work is devoted to the study of the impact of virtual objects in virtual, augmented and mixed reality systems on the quality of human visual perception. The features of the operation of virtual, augmented and mixed reality systems, the principles of image formation of the virtual and real worlds and the problems of their combination are described. For mixed reality systems, the basic principles of building systems, their specifics and the principles of forming an image of the real world are described. The specific reasons for the formation of the conflict of visual perception in mixed reality systems are considered. There are three main causes of conflict that can be overcome in modern mixed reality systems: the position and type of the light source; the correct formation of shadows from virtual sources on the image of real-world objects; and the intensity of radiation, including the radiation diagram. A method for evaluating the correctness of the formation of illumination of virtual objects in a mixed reality system and its influence on the quality of visual perception is proposed. To evaluate the proposed methodology, a number of test scenes with correct and incorrect lighting of virtual objects have been developed. The assessment of the quality of visual perception was carried out on a test group of 24 people with experience in computer graphics systems. The quality was evaluated on a good/bad scale. Based on the results of the expert assessment, a conclusion was formed that defines the requirement for building a lighting system for virtual objects in mixed reality systems.

show abstract

An Adaptive Algorithm for Motion Recognition of Throwing Arm on Account of Neural Network Technology

Ma¹,

Lù²

2023

Lecture Notes on Data Engineering and Communications Technologies

View full text Add to dashboard Cite

Restoration of Lighting Parameters in Mixed Reality Systems Using Convolutional Neural Network Technology Based on RGBD Images

Cited by 6 publications

References 16 publications

Image Synthesis Pipeline for CNN-Based Sensing Systems

Image Synthesis Pipeline for CNN-Based Sensing Systems

A method for evaluating the correctness of the illumination of virtual objects in mixed reality systems

An Adaptive Algorithm for Motion Recognition of Throwing Arm on Account of Neural Network Technology

Contact Info

Product

Resources

About