Neural Network Compression for Noisy Storage Devices

Berivan, Isik,; Choi, Kristy; Zheng, Xiaohui; Weissman, Tsachy; Ermon, Stefano; Wong, H.-S. Philip; Alaghi, Armin

doi:10.48550/arxiv.2102.07725

Cited by 4 publications

(6 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural Network Compression. The interest in neural network compression research has increased sharply in the last decade [11,13,15,26] as the growing size of neural networks becomes a significant bottleneck in their deployment on resource-constraint devices [6,20]. Pruning is one of the neural network compression techniques that has been improved to the extent that today it is possible to prune more than 90% of the parameters of a trained model without a performance degradation [9,11,12,16].…”

Section: Related Workmentioning

confidence: 99%

Neural 3D Scene Compression via Model Compression

Isik

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Rendering 3D scenes requires access to arbitrary viewpoints from the scene. Storage of such a 3D scene can be done in two ways; (1) storing 2D images taken from the 3D scene that can reconstruct the scene back through interpolations, or (2) storing a representation of the 3D scene itself that already encodes views from all directions. So far, traditional 3D compression methods have focused on the first type of storage and compressed the original 2D images with image compression techniques. With this approach, the user first decodes the stored 2D images and then renders the 3D scene. However, this separated procedure is inefficient since a large amount of 2D images have to be stored. In this work, we take a different approach and compress a functional representation of 3D scenes. In particular, we introduce a method to compress 3D scenes by compressing the neural networks that represent the scenes as neural radiance fields. Our method provides more efficient storage of 3D scenes since it does not store 2D images -which are redundant when we render the scene from the neural functional representation. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Neural 3D Scene Compression via Model Compression

Isik

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Also fortunately, it is likely that 32 bits per floating point parameter is an order of magnitude more than necessary. Prior work has shown that simple model compression can be performed at 8 bits per floating point parameter [10,77,86] or even more aggressively at 1-4 bits per floating point parameter [30,34,35,54,75,85,87] with very low loss in performance, even with coordinate based networks such as NeRF [13,33]. However, since model compression is outside the scope of our work, we simply parameterize our results by the number of bits per floating point parameter.…”

Section: Side Informationmentioning

confidence: 99%

LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks

Berivan¹,

Chou²,

Hwang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the attributes of a point cloud as samples of a vector-valued volumetric function at discrete positions. To compress the attributes given the positions, we compress the parameters of the volumetric function. We model the volumetric function by tiling space into blocks, and representing the function over each block by shifts of a coordinatebased, or implicit, neural network. Inputs to the network include both spatial coordinates and a latent vector per block. We represent the latent vectors using coefficients of the region-adaptive hierarchical transform (RAHT) used in the MPEG geometry-based point cloud codec G-PCC. The coefficients, which are highly compressible, are ratedistortion optimized by back-propagation through a ratedistortion Lagrangian loss in an auto-decoder configuration. The result outperforms RAHT by 2-4 dB. This is the first work to compress volumetric functions represented by local coordinate-based neural networks. As such, we expect it to be applicable beyond point clouds, for example to compression of high-resolution neural radiance fields.

show abstract

“…Substituting (11) into (10) suggests that minimizing MSE p is equivalent to minimizing MSE w with a modified prior Gaussian with mean µ λ and variance σ 2 λ (as opposed to µ p and σ 2 p ). Following (8), the estimate W that minimizes MSE p is then…”

Section: Bayesian Estimation With Compensatorsmentioning

confidence: 99%

“…These parameters are few in number. In practice, we can transmit/store them in a reliable manner (for example, via digital communication/storage [6] or protect them by repetition coding [11]).…”

Section: A Implementation Detailsmentioning

confidence: 99%

“…In particular, the noise in analog hardware is often modeled as AWGN to mimic the overall effect of many concurrent random effects. The efficient storage of NN weights in analog hardware, on the other hand, faces the same problem: the retrieved NN from an analog device is a noisy version of the stored NN [11], due to the noisy nature of analog hardware.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Denoising Noisy Neural Networks: A Bayesian Approach with Compensation

Shao

Liew

Gündüz

2021

Preprint

View full text Add to dashboard Cite

Noisy neural networks (NoisyNNs) refer to the inference and training of NNs in the presence of noise. Noise is inherent in most communication and storage systems; hence, NoisyNNs emerge in many new applications, including federated edge learning, where wireless devices collaboratively train a NN over a noisy wireless channel, or when NNs are implemented/stored in an analog storage medium. This paper studies a fundamental problem of NoisyNNs: how to estimate the uncontaminated NN weights from their noisy observations or manifestations. Whereas all prior works relied on the maximum likelihood (ML) estimation to maximize the likelihood function of the estimated NN weights, this paper demonstrates that the ML estimator is in general suboptimal. To overcome the suboptimality of the conventional ML estimator, we put forth an MMSE pb estimator to minimize a compensated mean squared error (MSE) with a population compensator and a bias compensator. Our approach works well for NoisyNNs arising in both 1) noisy inference, where noise is introduced only in the inference phase on the already-trained NN weights; and 2) noisy training, where noise is introduced over the course of training. Extensive experiments on the CIFAR-10 and SST-2 datasets with different NN architectures verify the significant performance gains of the MMSE pb estimator over the ML estimator when used to denoise the NoisyNN. For noisy inference, the average gains are up to 156% for a noisy ResNet34 model and 14.7% for a noisy BERT model; for noisy training, the average gains are up to 18.1 dB for a noisy ResNet18 model.

show abstract

Neural Network Compression for Noisy Storage Devices

Cited by 4 publications

References 23 publications

Neural 3D Scene Compression via Model Compression

Neural 3D Scene Compression via Model Compression

LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks

Denoising Noisy Neural Networks: A Bayesian Approach with Compensation

Contact Info

Product

Resources

About