The verification performance of biometric systems is normally evaluated using the receiver operating characteristic (ROC) or detection error trade-off (DET) curve. We propose two new ideas for statistical evaluation of biometric systems based on these data. The first is a new way to normalize match score distributions. A normalized match score,t, is calculated as a function of the angle from a representation of (FMR, FNMR) values in polar coordinates from some center. This has the advantage that it does not produce counterintuitive results for systems with unusual DET performance. Secondly, building on this normalization we develop a methodology to calculate an average DET curve. Each biometric system is represented in terms oft to allow genuine and impostor distributions to be combined, and an average DET is then calulated from these new distributions. We then show that this method is equivalent to direct averaging of DET data along each angle from the center. This procedure is then applied to data from a study of human matchers of facial images.

West Virginia University. His Ph.D. was in forest resource science (dendrochronology) from West Virginia University. His research interests include analytic methods in dendrochronology and forestry, nonlinear modeling, biased estimation, and working on applied statistics. J.J. COLBERT is a research mathematician with the Northeastern Research Station, USDA Forest Service. He received an M.S. in mathematics from Idaho State University and a Ph.D. in mathematics from Washington State University. His primary research interests include the modeling of forest ecosystem processes and integration of effects of exogenous inputs on forest stand dynamics.

Automatic face recognition (AFR) technologies have seen dramatic improvements in performance over the past decade, and such systems are now widely used for security and commercial applications. Since recognizing faces is a task that humans are understood to be very good at, it is common to question the relative performance of AFR and human testers. This paper addresses this question by: 1) conducting recognition tests on commercial AFR systems and human testers, and 2) developing statistical methods to compare the performance of different biometric matchers. Face recognition performance was tested by presenting face image pairs; humans were asked to respond on a scale of "Same", "Probably Same", "Not sure", "Probably Different", and "Different", while the biometric match score was measured from AFR systems. To evaluate these results, two new statistical evaluation techniques are developed. The first is a new way to normalize match score distributions, where a normalized match score,t, is calculated as a function of the angle from a representation of (FMR, FNMR) values in polar coordinates from some center. Using this normalization we develop a second methodology to calculate an average detection error trade-off (DET) curve, and show that this method is equivalent to direct averaging of DET data along each angle from the center. This procedure is then applied to compare the performance of the best AFR algorithms available to us in the years 1999, 2001, 2003, 2005 and 2006, in comparison to human scores. Results show algorithms have dramatically improved in performance over that time; in 2006, the ratio of human participants performing better than AFR to those performing worse is 0.78.

No abstract