RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion

Abstract

We present RL-GAN-Net, where a reinforcement learning (RL) agent provides fast and robust control of a generative adversarial network (GAN). Our framework is applied to point cloud shape completion that converts noisy, partial point cloud data into a high-fidelity completed shape by controlling the GAN. While a GAN is unstable and hard to train, we circumvent the problem by (1) training the GAN on the latent space representation whose dimension is reduced compared to the raw point cloud input and (2) using an RL agent to find the correct input to the GAN to generate the latent space representation of the shape that best fits the current input of incomplete point cloud. The suggested pipeline robustly completes point cloud with large missing regions. To the best of our knowledge, this is the first attempt to train an RL agent to control the GAN, which effectively learns the highly nonlinear mapping from the input noise of the GAN to the latent space of point cloud. The RL agent replaces the need for complex optimization and consequently makes our technique real time. Additionally, we demonstrate that our pipelines can be used to enhance the classification accuracy of point cloud with missing data.

Vaccine-style-net

“…They use latent optimization to find the best CLR, but the process of optimization is time-consuming. Then, RLGAN [10] uses a reinforcement learning agent to make the process real-time. The pipeline of direct methods is simpler than that of indirect methods, and because direct methods can directly optimize CD, so they usually achieve better quantitative results of CD.…”

“…3.1.1 Continuous Representation (CR) for 3D Geometry. Most existing data-driven point cloud completion methods [7,10,24] use an autoencoder (AE) or a variation of autoencoder that proposed by [1] to create a low-dimensional encoding of input point cloud. The autoencoder is a network that trained by directly reconstructing the input point cloud.…”

“…However, GANs [6] are successful at learning data representations and generating realistic samples from complex underlying distributions. Similar to [7,10], we also train a latent-space GAN, but our l-GAN is trained on our CLR that was generated from our pretrained CR network. The generator of our l-GAN synthesizes a new CLR ( ) from a noise , and the new CLR ( ) is regarded as the guidance of the classifier , which can then produce a complete 3D shape.…”

“…The environment is usually modeled as Markov decision process: the current state and action only depend on the previous state and action. We follow the assumption in [10]: the recent most observation is enough to define state . Corresponding to our setting, as shown in Fig.…”

“…Both methods focus on the existing local information without considering the relationship globally. With the development of point-cloud-based deep learning framework [16,17], a series of methods [10,18,24,26] have found remarkable results for completion tasks. They can infer the complete geometry from a single or a combination of partial views.…”