An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks

Abstract

It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized. Recently, Vershynin (2020) settled a long standing question by Baum (1988), proving that deep threshold networks can memorize n points inweights, where δ is the minimum distance between the points. In this work, we improve the dependence on δ from exponential to almost linear, proving that O( 1 δ + √ n) neurons and O( d δ + n) weights are sufficient. Our construction uses Gaussian random weights only in the first layer, while all the subsequent layers use binary or integer weights. We also prove new lower bounds by connecting memorization in neural networks to the purely geometric problem of separating n points on a sphere using hyperplanes.

On the Optimal Memorization Power of ReLU Neural Networks

“…[37] showed that threshold and ReLU networks can memorize N binary-labeled unit vectors in R d separated by a distance of δ > 0, using Õ e 1/δ 2 + √ N neurons and Õ e 1/δ 2 (d + √ N ) + N parameters. [27] improved the dependence on δ by giving a construction with Õ 1 δ + √ N neurons and Õ d δ + N parameters. This result holds only for threshold networks, but does not assume that the inputs are on the unit sphere.…”

“…Many works have shown results regarding the memorization power of neural networks, using different assumptions on the activation function and data samples (see e.g. [19,18,4,37,13,14,8,26,17,39,40,25,27,32]). The question of memorization also have practical implications on phenomenons such as "double descent" [5,24] which connects the memorization power of neural networks with their generalization capabilities.…”