Manifold Matching via Deep Metric Learning for Generative Modeling

Abstract

We propose a manifold matching approach to generative models which includes a distribution generator (or data generator) and a metric generator. In our framework, we view the real data set as some manifold embedded in a highdimensional Euclidean space. The distribution generator aims at generating samples that follow some distribution condensed around the real data manifold. It is achieved by matching two sets of points using their geometric shape descriptors, such as centroid and p-diameter, with learned distance metric; the metric generator utilizes both real data and generated samples to learn a distance metric which is close to some intrinsic geodesic distance on the real data manifold. The produced distance metric is further used for manifold matching. The two networks are learned simultaneously during the training process. We apply the approach on both unsupervised and supervised learning tasks: in unconditional image generation task, the proposed method obtains competitive results compared with existing generative models; in super-resolution task, we incorporate the framework in perception-based models and improve visual qualities by producing samples with more natural textures. Both theoretical analysis and real data experiments guarantee the feasibility and effectiveness of the proposed framework.

Implicit Data Augmentation Using Feature Interpolation for Low-Shot Image Generation

“…Compared to the input data dimension m = 1024×1024×3 and feature space dimension n = 50, the valid feature subspace is significantly flat and uniform. This experimental result is consistent with results in [11], even though the training settings being used are very different. An example of how to infer the shape of data set using eigenvalue curve is shown in Figure 3.…”

“…The key idea of this paper comes from the interesting flattening effect of discriminator observed in [11]. Particularly, in their paper [11], Dai and Hang interpret the discriminator as a metric generator which learns some intrinsic metric of real data manifold such that the manifold is flat under the learned metric. In this paper, we observe the similar behaviors in the geometric GAN [29] framework with hinge loss.…”

“…We utilize the following way introduced in [11,33] to detect the change of the learned metric along the training process: (i) For each iteration i, sample some real data points to form a finite metric space X i ; (ii) Construct the normalized distance matrix of X i under the learned metric d i ; (iii) Apply multidimensional scaling to M i to obtain the decreasing (finite) sequence of eigenvalues C i . Now we observe the eigenvalues to see how the Euclidean distance among feature vectors evolve during the training process.…”

“…The key question becomes how to reflect the degree of "flatness" of data manifold. As mentioned earlier, such information can be reached by the MDS of pairwise distances between features [11,33]. Smaller and more uniform eigenvalues of MDS indicate the shape of manifold is more "flat", where we tend to assign a higher p, otherwise we tend to use a smaller p to reduce the risk of introducing more bias from interpolation.…”

“…Their proposed SLE modules make it possible to generate high-resolution and visually appealing samples using only hundreds of samples. Dai and Hang [11] introduce a metric learning perspective to the GAN discriminator with multi-dimensional output and reveal an interesting flattening effect: along the training process, the learned metric gradually becomes more uniform and flat. These works inspire us to explore the possibility of implementing augmentation in feature space for few-shot data generation.…”