Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams

Abstract

Motivated by differentially-private (DP) training of machine learning models and other applications, we investigate the problem of computing prefix sums in the online (streaming) setting with DP. This problem has previously been addressed by special-purpose tree aggregation schemes with hand-crafted estimators. We show that these previous schemes can all be viewed as specific instances of a broad class of matrix-factorization-based DP mechanisms, and that in fact much better mechanisms exist in this class.In particular, we characterize optimal factorizations of linear queries under online constraints, deriving existence, uniqueness, and explicit expressions that allow us to efficiently compute optimal mechanisms, including for online prefix sums. These solutions improve over the existing state-ofthe-art by a significant constant factor, and avoid some of the artifacts introduced by the use of the tree data structure.

Constant matters: Fine-grained Complexity of Differentially Private Continual Observation

“…In a concurrent, independent work, McMahan et al [MRT22] used similar techniques of matrix factorization to show a bound in the expected 2 2 norm (equation 3 in their paper). In contrast, we give a bound in ∞ norm.…”

“…As a result, we do not have to solve a convex program, but compute the entries of the factorization using a recurrence relation (for M count ) and solving T(T + 1)/2 linear equations (for M average ). Finally, our explicit factorization for M count has a nice property that there are exactly T distinct entries (instead of possibly T 2 entries in McMahan et al [MRT22]) in the factorization. This has large impact on computation in practice.…”

“…The binary tree mechanism of Chan et al [CSS11] and Dwork et al [DNPR10] and its improvement by Honaker [Hon15] can be seen as a factorization. This has been independently noticed by McMahan et al [MRT22]. While Chan et al [CSS11] and Dwork et al [DNPR10] do allow computation on streaming data, Honaker's optimization [Hon15] does not because for a partial sum [1, t] it also uses the information stored at the nodes formed after time t. Therefore, for this comparison with related work, we do not discuss the Honaker's optimization [Hon15].…”

“…The most relevant work with ours is the concurrent work by McMahan et al [MRT22] that also looks at concrete bounds on performing counting under continual observation. The work of McMahan et al [MRT22] is motivated by performing optimization privately on streamed data. Therefore, they bound the expected mean squared error (i.e., in 2 2 norm) on privately computing a running sum.…”

“…However, it is also beneficial in the sense that it allows us to perform exact steps required in the mechanism while getting an explicit bound on the additive error of the mechanism. It is not clear that factorization in terms of lower triangular matrices is also necessary for continual observation; however, as pointed out by McMahan et al [MRT22], any factorization would not work for continual observation -Honaker's optimization of binary mechanism [Hon15] can be seen as a factorization but it cannot be used for continual observation (see [MRT22, Proposition 2.1]).…”