Deep Learning-based Polygonal Mesh Reconstruction from Point Clouds for Mixed Reality Visualization

Abstract

Mixed reality systems create new forms of interaction between the physical and digital world by overlaying digital elements on the physical environment or creating virtual environments with physical elements. A key component for mixed reality are 3D scanning systems that capture the shape and texture of objects or scenes in the form of point clouds. For effective use of these data in mixed reality applications, they are transformed into polygonal meshes suitable for rendering, animation, and interaction. The proposed method includes the application of ResNet blocks for feature extraction, the use of an autoencoder to obtain latent 3D shapes, and the definition of wall and ceiling geometry using bounding boxes. The method allows obtaining a complete 3D model of the scene from a point cloud using deep learning and geometric analysis. In this work, two datasets were used for training and experiments: ShapeNet and ScanNet. These datasets represent a large and diverse collection of three-dimensional objects and scanned scenes with detailed annotation.