Robustness of Neural Networks to Parameter Quantization

Abstract

Keywords: Neural Networks · Edge Computing · Parameter Quantization · Robustness · Satisfiability Modulo Theories Quantization, a commonly used technique to reduce the memory footprint of a neural network for edge computing, entails reducing the precision of the floating-point representation used for the parameters of the network. The impact of such rounding-off errors on the overall performance of the neural network is estimated using testing, which is not exhaustive and thus cannot be used to guarantee the safety of the model. We present a framework based on Satisfiability Modulo Theory (SMT) solvers to quantify the robustness of neural networks to parameter perturbation. To this end, we introduce notions of local and global robustness that capture the deviation in the confidence of class assignments due to parameter quantization. The robustness notions are then cast as instances of SMT problems and solved automatically using solvers, such as dReal. We demonstrate our framework on two simple Multi-Layer Perceptrons (MLP) that perform binary classification on a two-dimensional input. In addition to quantifying the robustness, we also show that Rectified Linear Unit activation results in higher robustness than linear activations for our MLPs.

Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators

“…Some works also consider quantizing activations [32], [34], [35] or gradients [36], [37], [38]. While works such as [14], [15], [16], [39] study the robustness of DNNs to quantization, the robustness of various quantization schemes against random bit errors has not been studied. This is in stark contrast to our findings that quantization impacts robustness significantly.…”

“…We address robustness against random and/or adversarial bit errors in three steps: First, we analyze the impact of fixed-point quantization schemes on bit error robustness. This has been neglected both in prior work on low-voltage DNN accelerators [4], [19] and in work on quantization robustness [14], [15], [16]. This yields our robust quantization (Sec.…”

“…We consider quantization-aware training [26], [27] using a generic, deterministic fixed-point quantization scheme commonly used in DNN accelerators [5]. However, we focus on the impact of quantization schemes on robustness against random bit errors, mostly neglected so far [14], [15], [16]. We find that quantization affects robustness significantly, even if accuracy is largely unaffected.…”

“…To improve DNN robustness to the induced random bit errors, we first consider the impact of fixed-point quantization on robustness. While prior work [14], [15], [16] studies robustness to quantization, the impact of random bit errors in quantized weights has not been considered so far. We find that the choice of quantization scheme has tremendous impact on robustness, even though accuracy is not affected.…”

Incremental Verification of Fixed-Point Implementations of Neural Networks

“…The typical approach consists in modeling an ANN and its corresponding verification properties, in SMT-LIB [6], using integer and real arithmetic theories, and then employ off-the-shelf SMT solvers to find property violations. Recently, Murthy et al [40] have used SMT to quantify neural-network robustness regarding parameter perturbation. Unfortunately, such verification schemes can not precisely capture issues that appear in implementations of ANNs, for two main reasons: (i) one can not model bit-level operations using the theory of integer and real arithmetic [14], and (ii) libraries, such as TensorFlow [21], often take advantage of available graphics processing units (GPUs) to explore the inherent parallelism of ANNs; the translation to GPUs can be problematic [42,45].…”