Instant recovery of shape from spectrum via latent space connections

Marin, Riccardo; Rampini, Arianna; Castellani, Umberto; Rodolà, Emanuele; Ovsjanikov, Maks; Melzi, Simone

doi:10.1109/3dv50981.2020.00022

Cited by 20 publications

(56 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By using this aspect that a generative model has a tractable prior distribution, many studies of missing data imputation have been conducted in various fields [27]- [29]. In addition, some approaches uses an encoding paradigm that is ranked per order of magnitude that makes the reconstruction with partial data automatic if the connection were to be dropped [30].…”

Section: Related Workmentioning

confidence: 99%

Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

For effective human-robot teaming, it is important for the robots to be able to share their visual perception with the human operators. In a harsh remote collaboration setting, data compression techniques such as autoencoder can be utilized to obtain and transmit the data in terms of latent variables in a compact form. In addition, to ensure real-time runtime performance even under unstable environments, an anytime estimation approach is desired that can reconstruct the full contents from incomplete information. In this context, we propose a method for imputation of latent variables whose elements are partially lost. To achieve the anytime property with only a few dimensions of variables, exploiting prior information of the category-level is essential. A prior distribution used in variational autoencoders is simply assumed to be isotropic Gaussian regardless of the labels of each training datapoint. This type of flattened prior makes it difficult to perform imputation from the category-level distributions. We overcome this limitation by exploiting a category-specific multi-modal prior distribution in the latent space. The missing elements of the partially transferred data can be sampled, by finding a specific modal according to the remaining elements. Since the method is designed to use partial elements for anytime estimation, it can also be applied for data over-compression. Based on the experiments on the ModelNet and Pascal3D datasets, the proposed approach shows consistently superior performance over autoencoder and variational autoencoder up to 70% data loss. The software is open source and is available from our repository 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…This paper is an extended version of the work presented in Marin et al (2020). Compared to the former version, our contribution is as follows: (i) We investigate different types of latent space, including those generated by an auto-encoder model as well as parametric spaces associated with morphable models, and study different parametrizations thereof; (ii) we include human bodies among the classes of analyzed shapes; (iii) we further develop the tools provided by our model for a meaningful exploration of the latent space, showing how the spectral prior contributes to the interpretability of latent codes, and enabling the disentanglement of intrinsic and extrinsic geometry as a novel application (Sect.…”

Section: Related Workmentioning

confidence: 99%

Spectral Shape Recovery and Analysis Via Data-driven Connections

et al. 2021

Self Cite

View full text Add to dashboard Cite

We introduce a novel learning-based method to recover shapes from their Laplacian spectra, based on establishing and exploring connections in a learned latent space. The core of our approach consists in a cycle-consistent module that maps between a learned latent space and sequences of eigenvalues. This module provides an efficient and effective link between the shape geometry, encoded in a latent vector, and its Laplacian spectrum. Our proposed data-driven approach replaces the need for ad-hoc regularizers required by prior methods, while providing more accurate results at a fraction of the computational cost. Moreover, these latent space connections enable novel applications for both analyzing and controlling the spectral properties of deformable shapes, especially in the context of a shape collection. Our learning model and the associated analysis apply without modifications across different dimensions (2D and 3D shapes alike), representations (meshes, contours and point clouds), nature of the latent space (generated by an auto-encoder or a parametric model), as well as across different shape classes, and admits arbitrary resolution of the input spectrum without affecting complexity. The increased flexibility allows us to address notoriously difficult tasks in 3D vision and geometry processing within a unified framework, including shape generation from spectrum, latent space exploration and analysis, mesh super-resolution, shape exploration, style transfer, spectrum estimation for point clouds, segmentation transfer and non-rigid shape matching.

show abstract

“…Closer to our method, some works [27,13] describe the instrinsic shape, or style, of a 3D model using the spectrum of its Laplace Beltrami operator (LBO). This is made possible by the observation that the LBO spectrum is invariant to pose deformation.…”

Section: Related Workmentioning

confidence: 99%

“…[2,25] or through a discrete correspondence map, as with style transfer e.g. [27]. Another approach is to assume that the human shape can be characterised independently of the pose, i.e.…”

Section: Introductionmentioning

confidence: 99%

Neural Human Deformation Transfer

Basset,

Boukhayma,

Wuhrer

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the problem of human deformation transfer, where the goal is to retarget poses between different characters. Traditional methods that tackle this problem require a clear definition of the pose, and use this definition to transfer poses between characters. In this work, we take a different approach and transform the identity of a character into a new identity without modifying the character's pose. This offers the advantage of not having to define equivalences between 3D human poses, which is not straightforward as poses tend to change depending on the identity of the character performing them, and as their meaning is highly contextual. To achieve the deformation transfer, we propose a neural encoder-decoder architecture where only identity information is encoded and where the decoder is conditioned on the pose. We use pose independent representations, such as isometryinvariant shape characteristics, to represent identity features. Our model uses these features to supervise the prediction of offsets from the deformed pose to the result of the transfer. We show experimentally that our method outperforms state-of-the-art methods both quantitatively and qualitatively, and generalises better to poses not seen during training. We also introduce a fine-tuning step that allows to obtain competitive results for extreme identities, and allows to transfer simple clothing.

show abstract

Instant recovery of shape from spectrum via latent space connections

Cited by 20 publications

References 42 publications

Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

Spectral Shape Recovery and Analysis Via Data-driven Connections

Neural Human Deformation Transfer

Contact Info

Product

Resources

About