Of Black Boxes, Instruments, and Experts: Testing the Validity of Forensic Science

Abstract

This paper argues that judges assessing the scientific validity and the legal admissibility of forensic science techniques ought to privilege testing over explanation. Their evaluation of reliability should be more concerned with whether the technique has been adequately validated by appropriate empirical testing than with whether the expert can offer an adequate description of the methods she uses, or satisfactorily explain her methodology or the theory from which her claims derive. This paper explores these issues within two specific contexts: latent fingerprint examination and the use of breath tests for the detection of alcohol. Especially in the forensic science arena, I suggest courts have often been seduced by superficially plausible explanations and descriptions of a technique or method, and permitted these to serve as a substitute for empirical testing. Thinking through these two examples illustrates both why evaluating the extent of testing should be the most important method by which courts assess reliability, and why, when other forms of explanatory evidence are readily available, we may nonetheless elect to make use of them. This paper suggests that these descriptions and explanations may at times usefully supplement evidence of testing, but should not generally be substituted for it. Finally, this paper embraces a kind of evidentiary pragmatism, in which the quantum of evidence required to establish legal reliability is determined not in the abstract, but in relation to the evidence that is, or ought to be, available as a result of reasonable research and investigation.

Accuracy and reliability of forensic latent fingerprint decisions

“…There has been discussion (24,26,27) regarding the appropriateness of using qualified conclusions in investigation or testimony. The effects of qualified conclusions could be assessed in this study, as "inconclusive with corresponding features" (SI Appendix, section 1.5).…”

“…23). Our study is based on a black box approach, evaluating the examiners' accuracy and consensus in making decisions rather than attempting to determine or dictate how those decisions are made (11,24).…”

Expertise in Fingerprint Identification

“…Remarkably, given that fingerprint examiners have testified in court for over one hundred years, there have been few experiments directly investigating the extent to which experts can correctly match fingerprints to one another, how competent and proficient fingerprint experts are, how examiners make their decisions, or the factors that affect performance (M. B. Thompson et al, 2013a;Saks & Koehler, 2005;Spinney, 2010b;Loftus & Cole, 2004;Vokey et al, 2009;Mnookin, 2008a) Vokey, Tangen, and Cole (2009) found that novices (with no prior experience with prints)…”

“…• Second, is the approach scientifically supported (Mnookin, 2008a) give an indication of the scientific strength of an approach.…”

An experimental study of blind proficiency tests in forensic science

“…Key proposals include independence from law enforcement (Giannelli 1997, NAS 2009), forensic experts for the defense (Giannelli 2004), evidence line ups (Miller 1984(Miller , 1987Risinger et al 2002), random, independent, multiple examinations (Koppl 2005a), "sequential unmasking" effected through a neutral "case manager" (Risinger et al 2002;Krane et al 2008), and blind proficiency tests (Jonakait 1991;Zabell 2005;Mnookin 2008). Several parties have offered relatively comprehensive suites of reforms (Thomson 1974;Saks et al 2001;Illinois 2002;Koppl 2005a;Justice Project 2008;NAS 2009).…”