Benchmarking Bayesian Improved Surname Geocoding Against Machine Learning Methods

Abstract

Bayesian Improved Surname Geocoding (BISG) is the most popular method for proxying race/ethnicity in voter registration files that do not contain it. This paper benchmarks BISG against a range of previously untested machine learning alternatives, using voter files with self-reported race/ethnicity from California, Florida, North Carolina, and Georgia. This analysis yields three key findings. First, when given the exact same inputs, BISG and machine learning perform similarly for estimating aggregate racial/ethnic composition. Second, machine learning outperforms BISG at individual classification of race/ethnicity. Third, the performance of all methods varies substantially across states. These results suggest that pre-trained machine learning models are preferable to BISG for individual classification. Furthermore, mixed results at the precinct level and across states underscore the need for researchers to empirically validate their chosen race/ethnicity proxy in their populations of interest.

Estimating Racial Disparities When Race is Not Observed

“…We observe this pattern as well in the validation study in Section 4. New methods continue to be developed that improve the calibration of BISG probabilities, including some machine learning methods based on labeled data (Imai et al, 2022;Argyle and Barber, 2022;Decter-Frain, 2022).…”

“…Finally, while the discussion in this section has been focused on the BISG methodology, the qualitative results and necessary assumptions carry over to other approaches which produce probabilistic predictions of individual race, such as those recently developed by Argyle and Barber (2022) and Decter-Frain (2022). Just as with BISG, well-calibrated probabilities are not generally sufficient to produce unbiased estimates of racial disparities using the outputs of these methods.…”