Understanding adaptive gradient clipping in DP‐SGD, empirically

Abstract

Differentially Private Stochastic Gradient Descent (DP-SGD) is a prime method for training machine learning models with rigorous privacy guarantees. Since its birth, DP-SGD has gained popularity and has been widely adopted in both academic and industrial research. One well-known challenge when using DP-SGD is how to improve utility while maintaining privacy. To this end, recently we have seen several proposals that clip the gradients with adaptive thresholds rather than a fixed one. Although each proposal comes with some theoretical justification, the theories often rely on strong assumptions and are not compatible with each other. It is hard to know whether they are good in practice and how good they are. In this paper, we investigate adaptive clipping in DP-SGD from an empirical perspective. With extensive experiments, we were able to gain some fresh insights and proposed two new adaptive clipping strategies based on them. We cross-compared the existing methods and our new strategies experimentally. Results showed that our strategies did provide a substantial improvement in model accuracy, and outperformed the state-of-the-art adaptive clipping methods consistently.

Adaptive synchronous strategy for distributed machine learning

“…SGD optimizer uses gradients from the aggregator or Async group to update the global model parameters based on the SGD algorithm. 46,47 The updated parameters are sent to corresponding workers by PS.…”

zkChain: A privacy‐preserving model based on zk‐SNARKs and hash chain for efficient transfer of assets

“…With the help of core technologies such as distributed ledgers, asymmetric encryption, smart contracts, and consensus mechanisms, blockchain can realize functions such as point-to-point, anonymity, traceability, and anti-tampering, ensuring the credibility and security of data in a distributed environment. 1,2 With the development of core technologies, a mature blockchain technology system has been formed and applied to Internet of Things (IoT), [3][4][5][6] digital finance, 7 edge computing, 8 artificial intelligence (AI), [9][10][11][12][13][14] supply chain management (SCM) 15 and other fields. However, privacy attacks are still the core problem hindering the popularization and application of blockchain.…”