Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

Abstract

For effective human-robot teaming, it is important for the robots to be able to share their visual perception with the human operators. In a harsh remote collaboration setting, data compression techniques such as autoencoder can be utilized to obtain and transmit the data in terms of latent variables in a compact form. In addition, to ensure real-time runtime performance even under unstable environments, an anytime estimation approach is desired that can reconstruct the full contents from incomplete information. In this context, we propose a method for imputation of latent variables whose elements are partially lost. To achieve the anytime property with only a few dimensions of variables, exploiting prior information of the category-level is essential. A prior distribution used in variational autoencoders is simply assumed to be isotropic Gaussian regardless of the labels of each training datapoint. This type of flattened prior makes it difficult to perform imputation from the category-level distributions. We overcome this limitation by exploiting a category-specific multi-modal prior distribution in the latent space. The missing elements of the partially transferred data can be sampled, by finding a specific modal according to the remaining elements. Since the method is designed to use partial elements for anytime estimation, it can also be applied for data over-compression. Based on the experiments on the ModelNet and Pascal3D datasets, the proposed approach shows consistently superior performance over autoencoder and variational autoencoder up to 70% data loss. The software is open source and is available from our repository 1 .

An approach for obtaining estimation of stability of large communication network taking into account its dependent paths

“…Another advantage of the proposed method is that it is an anytime algorithm [Melnikov et al, 2009;Melnikov and Sayfullina, 2013;Melnikov et al, 2018;Grandcolas and Pain-Barre, 2022;Yu and Oh, 2022]. Moreover, it provides for setting the precision with which the reliability assessment will be obtained.…”

A Comprehensive Review of Vision-Based 3D Reconstruction Methods

“…Multi-source heterogeneous information fusion (MSHIF) comprehensively utilizes information obtained from different sensors, such as radar [67], lidar, camera, ultrasound, infrared thermal imager [68], GPS [69], MRI [70], IMU, and V2X, to overcome the limitations of individual sensors and create a more comprehensive perception of the environment or target, thereby enhancing the accuracy of 3D reconstruction [71]. Yu proposed a multimodal 3D object reconstruction method based on variational autoencoders [72]. This method automatically determines the modality during training, which includes specific categories of information.…”

Image2Lego: Customized LEGO Set Generation from Images

“…Image-to-3D 3D model construction from 2D images of objects is an active research area [4,9,26]. For example, Lim et al [13] demonstrate an algorithm for modeling fine-pose of objects within captured 2D images and matching them to a set of 3D models.…”