Treatment of inconclusives in the AFTE range of conclusions

Abstract

In the past decade, and in response to the recommendations set forth by the National Research Council Committee on Identifying the Needs of the Forensic Sciences Community (2009), scientists have conducted several black-box studies that attempt to estimate the error rates of firearm examiners. Most of these studies have resulted in vanishingly small error rates, and at least one of them (D. P. Baldwin, S. J. Bajic, M. Morris, and D. Zamzow. A Study of False-Positive and False-Negative Error Rates in Cartridge Case Comparisons. Technical report, Ames Lab IA, Performing, Fort Belvoir, VA, April 2014.) was cited by the President’s Council of Advisors in Science and Technology (PCAST) during the Obama administration, as an example of a well-designed experiment. What has received little attention, however, is the actual calculation of error rates and in particular, the effect of inconclusive findings on those error estimates. The treatment of inconclusives in the assessment of errors has far-reaching implications in the legal system. Here, we revisit several black-box studies in the area of firearms examination, investigating their treatment of inconclusive results. It is clear that there are stark differences in the rate of inconclusive results in regions with different norms for training and reporting conclusions. More surprisingly, the rate of inconclusive decisions for materials from different sources is notably higher than the rate of inconclusive decisions for same-source materials in some regions. To mitigate the effects of this difference we propose a unifying approach to the calculation of error rates that is directly applicable in forensic laboratories and in legal settings.

How to make better forensic decisions

“…Given the perilous sequalae of a forensic classification error, the natural human tendency is to default on difficult problems. Doing so, however, creates a different set of concerns, which have tied the field in knots and raise serious questions about the utility of firearms evidence ( 33 – 37 ).…”

Shining a Light on Forensic Black-Box Studies

“…The decision to change the treatment of inconclusives was seemingly influenced, in part, by high error rates in ballistics (CTS). Treating inconclusives as "correct" is problematic (for more details, see Hofmann et al, 2021;Dorfman and Valliant, 2022). However, there is no agreement on how to handle inconclusives in analyses (see e.g., Weller and Morris, 2020).…”

Inconclusives, errors, and error rates in forensic firearms analysis:Three statistical perspectives

“…Hofmann et al [ 16 ] offer four distinct Options for looking at Inconclusives, give a rationale for each, and, for three of the options, calculate the corresponding error rates in several recent studies, including the original Ames study.…”

“…The error rates as calculated in Ames-FBI [ 10 , 12 ], and virtually all firearms studies, are mathematically equivalent to adopting Option 2 (“correct”). The reader is referred to Hofmann et al [ 16 ] for the error rates under the different options for a multitude of recent studies.…”