Are Commercial Face Detection Models as Biased as Academic Models?

Abstract

As facial recognition systems are deployed more widely, scholars and activists have studied their biases and harms. Audits are commonly used to accomplish this and compare the algorithmic facial recognition systems' performance against datasets with various metadata labels about the subjects of the images. Seminal works have found discrepancies in performance by gender expression, age, perceived race, skin type, etc. These studies and audits often examine algorithms which fall into two categories: academic models or commercial models. We present a detailed comparison between academic and commercial face detection systems, specifically examining robustness to noise. We find that state-of-the-art academic face detection models exhibit demographic disparities in their noise robustness, specifically by having statistically significant decreased performance on older individuals and those who present their gender in a masculine manner. When we compare the size of these disparities to that of commercial models, we conclude that commercial models -in contrast to their relatively larger development budget and industry-level fairness commitments -are always as biased or more biased than an academic model. Recently, Dooley et al. [18] performed the first benchmark of commercial face detection systems' robustness to natural noise. They found that each of Amazon Rekognition, Microsoft Azure, and * Equal contribution.

Deep Learning Based Multi Pose Human Face Matching System

“…Images Human Faces Accuracy AFW [44] 12880 400 97% FDDB [45] 13000 500 96% Pascal face [37] 201609 650 95% IJB-A [46] 22320 497 92% MALF [47] 51009 960 91% RFFD [48] 393703 1680-1750 99.64%…”