Composite Shape Modeling via Latent Space Factorization

Abstract

We present a novel neural network architecture, termed Decomposer-Composer, for semantic structure-aware 3D shape modeling. Our method utilizes an auto-encoderbased pipeline, and produces a novel factorized shape latent space, where the semantic structure of the shape collection translates into a data-dependent sub-space factorization, and where shape composition and decomposition become simple linear operations on the embedding coordinates. We further propose to model shape assembly using an explicit learned part deformation module, which utilizes a 3D spatial transformer network to perform an innetwork volumetric grid deformation, and which allows us to train the whole system end-to-end. The resulting network allows us to perform part-level shape manipulation, unattainable by existing approaches. Our extensive ablation study, comparison to baseline methods and qualitative analysis demonstrate the improved performance of the proposed method.

Sp-Gan

“…In general, 3D shapes can be generated in a part-conditioned or unconditioned manner. Part-conditioned means that we employ an additional dataset with parts labels to train a part-wise variational autoencoder to encode each predefined (or learned) part into a latent distribution [Dubrovina et al 2019;Mo et al 2019Mo et al , 2020Wu et al 2020], so that new shapes can be generated by sampling from the distributions of parts and composing them. On the other hand, unconditioned means that we directly synthesize 3D shapes from a random distribution, so the generation process has full freedom and the generated samples are not limited by the part-annotation data or any pre-defined structure relation.…”

LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies

“…[11,17,20] introduced frameworks that assigned individual probability distributions to parts of shapes following flattened hierarchies. Similarly, the latent codes learnt in [5] were factored according to individual components of shapes, and [30] introduced a Seq2Seq model for flattened object hierarchies.…”

Synthesizing Human Faces using Latent Space Factorization and Local Weights (Extended Version)

“…They proved that the part-based model encourages the generator to create new data unseen in the training set. Dubrovina et al [4] handled the composition and decomposition of each part as a simple linear operation on the factorized embedding space. They used projection matrices to split full object encodings into part encodings.…”

“…To composite each part, both [3] and [4] compute per-part affine transformation. Our model does not utilize spatial transformer networks [13] nor computes affine transformation to combinate each part of the data.…”

“…Previous part-based generative models exploit explicit segmentation data or labels for training their model to learn the structure of the object parts or use several part decoders[2, 3,4]. However, existing 3D facial mesh datasets barely have pre-segmented data.…”