Neural 3D Scene Compression via Model Compression

Abstract

Rendering 3D scenes requires access to arbitrary viewpoints from the scene. Storage of such a 3D scene can be done in two ways; (1) storing 2D images taken from the 3D scene that can reconstruct the scene back through interpolations, or (2) storing a representation of the 3D scene itself that already encodes views from all directions. So far, traditional 3D compression methods have focused on the first type of storage and compressed the original 2D images with image compression techniques. With this approach, the user first decodes the stored 2D images and then renders the 3D scene. However, this separated procedure is inefficient since a large amount of 2D images have to be stored. In this work, we take a different approach and compress a functional representation of 3D scenes. In particular, we introduce a method to compress 3D scenes by compressing the neural networks that represent the scenes as neural radiance fields. Our method provides more efficient storage of 3D scenes since it does not store 2D images -which are redundant when we render the scene from the neural functional representation. 1

LVAC: Learned volumetric attribute compression for point clouds using coordinate based networks

“…Hence the focus of this paper is on compression of Z. We assume that the simple CBN parameterized by θ can be compressed using model compression tools, e.g., (Bird et al, 2021;Isik, 2021), to a few bits per parameter with little loss in performance. Alternatively, we assume that the CBN may be trained to generalize across many point clouds, obviating the need to transmit θ.…”

“…However, while there has been an explosion of work using CBNs for representing specific objects and scenes (Park et al, 2019a;Mescheder et al, 2019;Mildenhall et al, 2020;Sitzmann et al, 2020;Yu A. et al, 2021;Barron et al, 2021;Hedman et al, 2021;Knodt et al, 2021;Martel et al, 2021;Srinivasan et al, 2021;Takikawa et al, 2021;Zhang et al, 2021), none of that work focuses on compressing those representations. (Two exceptions may be (Bird et al, 2021;Isik, 2021), which simply apply model compression to the CBNs.). Good lossy compression is nontrivial, and must make the optimal trade-off between the fidelity of the reconstruction and the number of bits used in its binary representation.…”

LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks

“…Hence the focus of this paper is on compression of Z. We assume that the simple CBN parameterized by θ can be compressed using model compression tools, e.g., [13,33], to a few bits per parameter with little loss in performance. Alternatively, we assume that the CBN may be trained to generalize across point clouds, obviating the need to transmit θ.…”

“…Also fortunately, it is likely that 32 bits per floating point parameter is an order of magnitude more than necessary. Prior work has shown that simple model compression can be performed at 8 bits per floating point parameter [10,77,86] or even more aggressively at 1-4 bits per floating point parameter [30,34,35,54,75,85,87] with very low loss in performance, even with coordinate based networks such as NeRF [13,33]. However, since model compression is outside the scope of our work, we simply parameterize our results by the number of bits per floating point parameter.…”

“…However, while there has been an explosion of work using CBNs for representing specific objects and scenes, none of that work focuses on compressing those representations. (Two exceptions are [13,33], which apply model compression to the CBNs themselves.) Good lossy compression is nontrivial, and must make the optimal trade-off between the fidelity of the reconstruction and the number of bits used in its binary representation.…”