Comment: The Essential Role of Policy Evaluation for the 2020 Census DisclosureAvoidance System

Abstract

In "Differential Perspectives: Epistemic Disconnects Surrounding the U.S. Census Bureau's Use of Differential Privacy," boyd and Sarathy argue that empirical evaluations of the Census Disclosure Avoidance System (DAS), including our published analysis (Kenny et al., 2021b), failed to recognize that the benchmark data against which the 2020 DAS was evaluated is never a ground truth of population counts. In this commentary, we explain why policy evaluation, which was the main goal of our analysis, is still meaningful without access to a perfect ground truth. We also point out that our evaluation leveraged features specific to the decennial census and redistricting data, such as block-level population invariance under swapping and voter file racial identification, better approximating a comparison with the ground truth. Lastly, we show that accurate statistical predictions of individual race based on the Bayesian Improved Surname Geocoding, while not a violation of differential privacy, substantially increases the disclosure risk of private information the Census Bureau sought to protect. We conclude by arguing that policymakers must confront a key trade-off between data utility and privacy protection, and an epistemic disconnect alone is insufficient to explain disagreements between policy choices.

Evaluating bias and noise induced by the U.S. Census Bureau’s privacy protection methods

“…These past comparisons remain useful for policy evaluation and allow researchers to compare how policy outcomes may change when basing decisions on a new dataset. However, they could not assess the relative magnitude of bias and noise induced by the TopDown and swapping algorithms because their analyses depend on comparisons between versions of the DAS (25,29).…”

“…Together, these results provide additional empirical context for ongoing conversations about the Bureau's trade-off between data accuracy and privacy protection. Biases from data produced by the TopDown and swapping algorithms are smaller than found in the released NMF, but the extent to which privacy is meaningfully protected is disputed and dependent on the definition of privacy used (9,25). Further research and discussion about the privacy-accuracy trade-off and the real-world privacy protections offered by a formally private system such as TopDown are warranted.…”

“…The Bureau's particular swapping algorithm preserved a geographic unit's total population from the CEF, because it swapped households with the "same number of adults [and] same number of children" [(24), p. 518]. Therefore, counts for total population released after swapping serves as the ground truth against which we measure TopDown algorithm's error exactly (25). No statistical assumptions are necessary to compute this error.…”

Census officials must constructively engage with independent evaluations

“…Like the advocates of differential privacy, including Jarmin et al and Bun et al ( 2 ), we also believe that any DAS should make statistical inference possible. The key question is how to balance data accuracy and privacy protection ( 3 ).…”

“…Unlike Jarmin et al, who make theoretical arguments, we have focused on empirical evaluations of the 2020 DAS ( 3 – 6 ). Independent evaluations are essential for improvements of the DAS, as evidenced by previous work that identified issues with earlier implementations.…”