Tm-Net

Abstract

We introduce TM-NET, a novel deep generative model for synthesizing textured meshes in a part-aware manner. Once trained, the network can generate novel textured meshes from scratch or predict textures for a given 3D mesh, without image guidance. Plausible and diverse textures can be generated for the same mesh part, while texture compatibility between parts in the same shape is achieved via conditional generation. Specifically, our method produces texture maps for individual shape parts, each as a deformable box, leading to a natural UV map with limited distortion. The network separately embeds part geometry (via a PartVAE) and part texture (via a TextureVAE) into their respective latent spaces, so as to facilitate learning texture probability distributions conditioned on geometry. We introduce a conditional autoregressive model for texture generation, which can be conditioned on both part geometry and textures already generated for other parts to achieve texture compatibility. To produce high-frequency texture details, our Tex-tureVAE operates in a high-dimensional latent space via dictionary-based vector quantization. We also exploit transparencies in the texture as an effective means to model complex shape structures including topological details. Extensive experiments demonstrate the plausibility, quality, and diversity of the textures and geometries generated by our network, while avoiding inconsistency issues that are common to novel view synthesis methods. CCS Concepts: • Computing methodologies → Shape modeling.

Surface‐aware Mesh Texture Synthesis with Pre‐trained 2D CNNs

“…Among these, two approaches have been considered: Using a diffusive approach related to heat diffusion on surfaces [SACO22, WNEH22, MBM*17, MBBV15], or equivariant convolutions on surfaces designed to address the rotation ambiguity problem of the tangent plane [PO18, YLP*20, MKK21]. Lastly, texture‐based approaches have emerged as a viable solution, as textures enable the definition of data on a finer level than just the pure geometry of a mesh [HZY*19, LLZ*19, GWY*21]. All these approaches have been designed for a supervised setup where the downstream task is performed directly on the mesh surface.…”

Evaluation Metrics for Intelligent Generation of Graphical Game Assets: A Systematic Survey-Based Framework

“…These include object placement algorithms, part-wise synthesis, methods for interpolating between existing assets, style transfer and parametric systems. At a high-level, we group together individual 2D assets as sprites and maps, where sprites consist of graphics that may be used as characters [17] or objects [18] in a 2D world or user interface [13], and maps consist of texture [19], normal [20] or height-maps [21] that are commonly applied in improving the rendering of mesh based 3D assets. 3D assets may be arranged in an interior context, such as room layouts [22] or in an exterior context such as a city made up of buildings [23].…”

“…However, this is not always a complete limitation. For example, with variational autoencoders (VAEs), such metrics can validate how well the method captures the desired features of a dataset, with the method itself still being capable of producing novel and varied outputs by sampling or interpolating between data points [19], [44], [56]. Many games follow a specific art direction, with requirements for the style of assets.…”

“…Structural similarity (SSIM) measures the perceptual quality based on high-level structure, comparing the output to a ground-truth image [94]. This has seen widespread use across image and texture generation approaches [19], [38], [47], [92]. Yadav et al [95] use the visual information fidelity (VIF) metric [96] to assess the visual quality of their outputs.…”

Texture Inpainting for Photogrammetric Models

“…TM‐Net [GWY*21] is a generative model that has been recently proposed to synthesize 3D textured objects. The model produces objects assembled from ‘parts’ modeled as deformed squares tessellated at low resolution, with the texture data synthesized by a separate network and to a fixed UV layout.…”