Efficiently Learning Any One Hidden Layer ReLU Network From Queries

Abstract

Model extraction attacks have renewed interest in the classic problem of learning neural networks from queries. This work gives the first polynomial-time algorithm for learning one hidden layer neural networks provided black-box access to the network. Formally, we show that if F is an arbitrary one hidden layer neural network with ReLU activations, there is an algorithm with query complexity and running time that is polynomial in all parameters that outputs a network F ′ achieving low square loss relative to F with respect to the Gaussian measure. While a number of works in the security literature have proposed and empirically demonstrated the effectiveness of certain algorithms for this problem, ours is the first with fully polynomial-time guarantees of efficiency for worst-case networks (in particular our algorithm succeeds in the overparameterized setting).

Learning Deep ReLU Networks Is Fixed-Parameter Tractable

“…Since in a one-hidden-layer ReLU network (without bias term), F (0) = 0, and furthermore F is a 2kR-Lipschitz continuous function, we conclude that |F (x)| 2kR • x 2 for ∀x ∈ R d . Next, we can apply the proof of Lemma A.1 of [CKM21] to show that G(x) := F (x) 2 − µ is a zero-centered, sub-exponential random variable with sub-exponential norm G Ψ 1 = O(µ + 4R 2 k 3 ). Finally, by using the concentration property of sub-exponential random variables, we conclude that:…”