Inconclusives, errors, and error rates in forensic firearms analysis:Three statistical perspectives

Dorfman, Alan H.; Valliant, Richard

doi:10.1016/j.fsisyn.2022.100273

Cited by 19 publications

(29 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These concerns are not limited to just the PCAST. More recently, controversy has ensued as it relates to the methodological design and calculation of error rates, specifically how 'inconclusive' decisions are accounted for in error rate studies [8][9][10][11][12][13][14][15]. These concerns have raised questions as to whether courts and the public have been misled when presented with statistics touting the reliability of forensic evidence presented in courts.…”

Section: Forensic Methods In Questionmentioning

confidence: 99%

Registered reports in forensic science

Houck

Chin

Swofford³

et al. 2022

R. Soc. open sci.

View full text Add to dashboard Cite

Research assessing the validity and reliability of many forensic science disciplines has been published; however, the quality of this research varies depending on the methodologies employed. This was a major point of contention with the United States' President's Council of Advisors on Science and Technology, who recognized the existing literature but found the majority lacking because of methodological issues. Questionable scientific methodologies have undermined the forensic science community's ability to defend the scientific foundations and examination protocols used to examine evidence in criminal cases. Such scientific failures have significant legal implications. Registered reports, which strengthen the quality of scientific research and reliability of laboratory protocols, can provide transparency, validity and a stronger scientific foundation for forensic science.

show abstract

Section: Forensic Methods In Questionmentioning

confidence: 99%

Registered reports in forensic science

Houck

Chin

Swofford³

et al. 2022

R. Soc. open sci.

View full text Add to dashboard Cite

show abstract

“…Additionally, almost all black-box studies count inconclusives as ‘correct’ decisions. This is known to be problematic [ 19 , 20 ]. Hofmann et al [ 20 ] found that inconclusives were used at a much higher rate for different source items than same source items for firearms examiners, while Dorfman & Valliant [ 19 ] explored how the presence of inconclusives can negatively influence error rate estimates.…”

Section: Black-box Studiesmentioning

confidence: 99%

“…This is known to be problematic [ 19 , 20 ]. Hofmann et al [ 20 ] found that inconclusives were used at a much higher rate for different source items than same source items for firearms examiners, while Dorfman & Valliant [ 19 ] explored how the presence of inconclusives can negatively influence error rate estimates. However, treating inconclusives as correct remains the prevalent practice.…”

Section: Black-box Studiesmentioning

confidence: 99%

Hierarchical Bayesian non-response models for error rates in forensic black-box studies

Khan

Carriquiry

2023

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

Forensic science plays a critical role in the United States criminal legal system. Historically, however, most feature-based fields of forensic science, including firearms examination and latent print analysis, have not been shown to be scientifically valid. Recently, black-box studies have been proposed as a means of assessing whether these feature-based disciplines are valid, at least in terms of accuracy, reproducibility and repeatability. In these studies, forensic examiners frequently either do not respond to every test item or select an answer equivalent to ‘don’t know’. Current black-box studies do not account for these high levels of missingness in statistical analyses. Unfortunately, the authors of black-box studies typically do not share the data necessary to meaningfully adjust estimates for the high proportion of missing responses. Borrowing from work in the context of small area estimation, we propose the use of hierarchical Bayesian models that do not require auxiliary data to adjust for non-response. Using these models, we offer the first formal exploration of the impact that missingness is playing in error rate estimations reported in black-box studies. We show that error rates currently reported as low as 0.4% could actually be at least 8.4% in models accounting for non-response where inconclusive decisions are counted as correct, and over 28% when inconclusives are counted as missing responses. These proposed models are not the answer to the missingness problem in black-box studies. But with the release of auxiliary information, they can be the foundation for new methodologies to adjust for missingness in error rate estimations. This article is part of the theme issue ‘Bayesian inference: challenges, perspectives, and prospects’.

show abstract

“…But if their results are, as those authors suggest, to "offer additional resources to the courts as they weigh the admissibility and value of firearms testimony," then further exploration and discussion of the raw data concerning poor-performing examiners is warranted. Specifically, this commentary extracts and analyzes conclusions drawn during different-source comparisons because they pose the greatest danger of provoking wrongful convictions, not just by falsely implicating a particular gun (and by extension defendant) as associated with a criminal offense via false positives [2], but also by otherwise (generally through an inconclusive finding) failing to provide appropriately exculpatory evidence to defendants excluding a particular firearm as the source of crime-related bullets and/or cartridge cases [3][4][5]. As the authors of Monson et al themselves acknowledge, the question of how to calculate error rates for firearms examination given the existence of inconclusive decisions has sparked substantial controversy in recent years, with proposals to count inconclusive decisions (functionally) as correct [1], treat inconclusive decisions (functionally) as incorrect [4,6], or remove inconclusive decisions entirely from the calculation [2,7], all receiving support.…”

Section: E T T E R T O T H E E D I T Omentioning

confidence: 99%