Adam in Private: Secure and Fast Training of Deep Neural Networks with Adaptive Moment Estimation

Abstract

Machine Learning (ML) algorithms, especially deep neural networks (DNN), have proven themselves to be extremely useful tools for data analysis, and are increasingly being deployed in systems operating on sensitive data, such as recommendation systems, banking fraud detection, and healthcare systems. This underscores the need for privacy-preserving ML (PPML) systems, and has inspired a line of research into how such systems can be constructed efficiently. However, most prior works on PPML achieve efficiency by requiring advanced ML algorithms to be simplified or substituted with approximated variants that are “MPC-friendly” before multi-party computation (MPC) techniques are applied to obtain a PPML systems. A drawback of this approach is that it requires careful fine-tuning of the combined ML and MPC algorithms, and might lead to less efficient algorithms or inferior quality ML (such as lower prediction accuracy). This is an issue for secure training of DNNs in particular, as this involves several arithmetic algorithms that are thought to be “MPCunfriendly”, namely, integer division, exponentiation, inversion, and square root extraction. In this work, we take a structurally different approach and propose a framework that allows efficient and secure evaluation of full-fledged state-of-the-art ML algorithms via secure multi-party computation. Specifically, we propose secure and efficient protocols for the above seemingly MPC-unfriendly computations (but which are essential to DNN). Our protocols are three-party protocols in the honest-majority setting, and we propose both passively secure and actively secure with abort variants. A notable feature of our protocols is that they simultaneously provide high accuracy and efficiency. This framework enables us to efficiently and securely compute modern ML algorithms such as Adam (Adaptive moment estimation) and the softmax function “as is”, without resorting to approximations. As a result, we obtain secure DNN training that outperforms state-of-the-art threeparty systems; our full training is up to 6.7 times faster than just the online phase of FALCON (Wagh et al. at PETS’21) and up to 4.2 times faster than Dalskov et al. (USENIX’21) on the standard benchmark network for secure training of DNNs. The potential advantage of our approach is even greater when considering more complex realistic networks. To demonstrate this, we perform measurements on real-world DNNs, AlexNet and VGG16, which are large networks containing millions of parameters. The performance of our framework for these networks is up to a factor of 26 ∼ 33 faster for AlexNet and 48 ∼ 51 faster for VGG16 to achieve an accuracy of 60% and 70%, respectively, when compared to FALCON. Even compared to CRYPTGPU (Tan et al. IEEE S&P’21), which is optimized for and runs on powerful GPUs, our framework achieves a factor of 2.1 and 4.1 faster performance, respectively, on these networks.

Efficient optimisation framework for convolutional neural networks with secure multiparty computation

“…This improves upon comparable state-of-the-art frameworks including ABY 3 [16], which achieved 94% accuracy in 2700 seconds, and FALCON [17], which reached the same accuracy in 780 seconds (online phase only). The aforementioned frameworks also converged after 15 training epochs, while Attrapadung et al [18] achieved convergence after just one epoch.…”

“…A recent study from Attrapadung et al [18] focuses on developing secure and efficient protocols for computing the Adam (Adaptive moment estimation) optimisation algorithm. Optimisation algorithms may be added to a neural network architecture to improve the speed of convergence during training.…”

Threats, Vulnerabilities, and Controls of Machine Learning Based Systems: A Survey and Taxonomy

“…We do not address the details of conventional threats of model/data thefts, which may exploit vulnerabilities in the development software, the development environment, the conventional software components in the system, the computing environment, and the operating organization. As a measure to prevent conventional data theft, secure multi-party computation is a technology for computing encrypted data without decrypting them, and has been actively studied for its applications to machine learning [56][57][58].…”

Klasifikasi Media Pembuangan Sampah Menggunakan Metode Resnet101-V2

“…Kemudian masuk ke tahap feature learning yang menggunakan lapisan konvolusi dengan aktivasi ReLU [12], MaxPooling, dan lapisan Fully Connected. Dari hasil lapisan Fully Connected tersebut akan dilakukan perubahan learning rate optimizer Adam [13] yang berbeda dan akan masuk ke metode Resnet-101v2 untuk menghitung hasil prediksi yang akan dijadikan accuracy, precision, dan recall. Metode Resnet-101v2 tersebut digunakan metode Skip Connection [14] yang akan membantu menentukan parameter dan pengulangan yang tepat agar dapat menghasilkan hasil prediksi dari arsitektur tersebut.…”