LatentKeypointGAN: Controlling GANs via Latent Keypoints

He, Xingzhe; Wandt, Bastian; Rhodin, Helge

doi:10.48550/arxiv.2103.15812

Cited by 4 publications

(9 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, keypoints detection has lately also being used to tackle other problems such as improving the quality of image generation [18] or object detection by estimating the object's center as keypoint [19]. We use keypoints to describe the grasp manifolds and estimate them during inference.…”

Section: Related Workmentioning

confidence: 99%

GraspME -- Grasp Manifold Estimator

Hager¹,

Bauer²,

Toussaint³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we introduce a Grasp Manifold Estimator (GraspME) to detect grasp affordances for objects directly in 2D camera images. To perform manipulation tasks autonomously it is crucial for robots to have such graspability models of the surrounding objects. Grasp manifolds have the advantage of providing continuously infinitely many grasps, which is not the case when using other grasp representations such as predefined grasp points. For instance, this property can be leveraged in motion optimization to define goal sets as implicit surface constraints in the robot configuration space. In this work, we restrict ourselves to the case of estimating possible end-effector positions directly from 2D camera images. To this extend, we define grasp manifolds via a set of keypoints and locate them in images using a Mask R-CNN [1] backbone. Using learned features allows to generalize to different view angle, with potentially noisy images, and objects that were not part of the training set. We rely on simulation data only and perform experiments on simple and complex objects, including unseen ones. Our framework achieves an inference speed of 11.5 fps on a GPU, an average precision for keypoint estimation of 94.5% and a mean pixel distance of only 1.29. This shows that we can estimate the objects very well via bounding boxes and segmentation masks as well as approximate the correct grasp manifold's keypoint coordinates.

show abstract

Section: Related Workmentioning

confidence: 99%

GraspME -- Grasp Manifold Estimator

Hager¹,

Bauer²,

Toussaint³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Keypoints must follow the transformation that is applied to the original images-equivariance. Recently, [13] introduced an alternative. They use a GAN to generate images along with corresponding latent keypoints and use them to train a detector.…”

Section: Related Workmentioning

confidence: 99%

“…Discovering object parts in images is a fundamental problem in computer vision as parts provide an intermediate representation that is robust to object appearance and pose variations [15,46]. Many high-level tasks benefit from part representations, such as 3D reconstruction [29,63], pose estimation [25,40], and image editing [13,62]. Keypoints and part segmentation maps are among the most commonly used forms.…”

Section: Introductionmentioning

confidence: 99%

“…The key question we address is how to design a generative adversarial network (GAN) that genertates images with part segmentation masks that are meaningful, i.e., group pixels into regions that typically move together and have shared appearance across images. We start from a backbone architecture that is borrowed from supervised segmentation networks [3,41] and the GAN strategy is inspired by its recent application to unsupervised keypoint detection [13]. Our innovation is the hierarchical generation of the image via multiple abstraction levels, including the use of masks.…”

Section: Introductionmentioning

confidence: 99%

“…In the first level, we use Gaussian noise to generate part appearance embeddings and a set of 2D latent points. Unlike [13] which continues straight from points to image generation, we first group points to define the position and the scale of each part. In the second abstraction level, we use a part-relative positional encoding to generate 2D feature maps and then generate the mask with a CNN.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation

He¹,

Wandt²,

Rhodin³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Segmenting an image into its parts is a common preprocess for high-level vision tasks such as image editing. However, annotating masks for supervised training is expensive. Weakly-supervised and unsupervised methods exist, but they depend on the comparison of pairs of images, such as from multi-views, frames of videos, and image transformations of single images, which limit their applicability. To address this, we propose a GAN-based approach that generates images conditioned on latent masks, thereby alleviating full or weak annotations required by previous approaches. We show that such mask-conditioned image generation can be learned faithfully when conditioning the masks in a hierarchical manner on 2D latent points that define the position of parts explicitly. Without requiring supervision of masks or points, this strategy increases robustness of mask to viewpoint and object position changes. It also lets us generate image-mask pairs for training a segmentation network, which outperforms state-of-the-art unsupervised segmentation methods on established benchmarks.

show abstract

Improving GAN Equilibrium by Raising Spatial Awareness

Wang¹,

Yang²,

Xu³

et al. 2021

Preprint

View full text Add to dashboard Cite

The success of Generative Adversarial Networks (GANs) is largely built upon the adversarial training between a generator (G) and a discriminator (D). They are expected to reach a certain equilibrium where D cannot distinguish the generated images from the real ones. However, in practice it is difficult to achieve such an equilibrium in GAN training, instead, D almost always surpasses G. We attribute this phenomenon to the information asymmetry between D and G. Specifically, we observe that D learns its own visual attention when determining whether an image is real or fake, but G has no explicit clue on which regions to focus on for a particular synthesis. To alleviate the issue of D dominating the competition in GANs, we aim to raise the spatial awareness of G. Randomly sampled multi-level heatmaps are encoded into the intermediate layers of G as an inductive bias. Thus G can purposefully improve the synthesis of certain image regions. We further propose to align the spatial awareness of G with the attention map induced from D. Through this way we effectively lessen the information gap between D and G. Extensive results show that our method pushes the two-player game in GANs closer to the equilibrium, leading to a better synthesis performance. As a byproduct, the introduced spatial awareness facilitates interactive editing over the output synthesis. Demo video and more results are at https://genforce.github.io/eqgan/.

show abstract

LatentKeypointGAN: Controlling GANs via Latent Keypoints

Cited by 4 publications

References 78 publications

GraspME -- Grasp Manifold Estimator

GraspME -- Grasp Manifold Estimator

GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation

Improving GAN Equilibrium by Raising Spatial Awareness

Contact Info

Product

Resources

About