Signal Detectability and Medical Decision-Making

Lusted, Lee B.

doi:10.1126/science.171.3977.1217

Cited by 278 publications

(140 citation statements)

References 15 publications

Supporting

Mentioning

134

Contrasting

Unclassified

Order By: Relevance

“…ROC are based on statistical decision theory and were introduced in the 1950s as reported by Metz [27] in the context of electronic signal detection and interpretation problems with radar. Later, the ROC were applied to medical imaging [28,29] and pap smear analysis [30]. The calculation of ROC for accuracy measurement is so widely used within the ÿeld of biomedical engineering, that its application has become a worldwide standard.…”

Section: Accuracy Evaluationmentioning

confidence: 99%

Human vs. machine: evaluation of fluorescence micrographs

Nattkemper

Twellmann

Ritter

et al. 2003

Computers in Biology and Medicine

View full text Add to dashboard Cite

To enable high-throughput screening of molecular phenotypes, multi-parameter uorescence microscopy is applied. Object of our study is lymphocytes which invade human tissue. One important basis for our collaborative project is the development of methods for automatic and accurate evaluation of uorescence micrographs. As a part of this, we focus on the question of how to measure the accuracy of microscope image interpretation, by human experts or a computer system. Following standard practice we use methods motivated by receiver operator characteristics to discuss the accuracies of human experts and of neural network-based algorithms. For images of good quality the algorithms achieve the accuracy of the medium-skilled experts. In images with increased noise, the classiÿers are outperformed by some of the experts. Furthermore, the neural network-based cell detection is much faster than the human experts. ?

show abstract

Section: Accuracy Evaluationmentioning

confidence: 99%

Human vs. machine: evaluation of fluorescence micrographs

Nattkemper

Twellmann

Ritter

et al. 2003

Computers in Biology and Medicine

View full text Add to dashboard Cite

show abstract

“…With regard to lineup format (simultaneous vs. sequential lineups), recent mock crime studies using receiver operating characteristic (ROC) analysis (15)(16)(17) have generally found that simultaneous lineups are, if anything, diagnostically superior to sequential lineups (18)(19)(20)(21). Similarly, in a recent police department field study comparing the two lineup formats, expert ratings of evidence against identified suspects favored the simultaneous procedure (22).…”

mentioning

confidence: 99%

Estimating the reliability of eyewitness identifications from police lineups

Wixted

Mickes

Dunn

et al. 2015

Proc. Natl. Acad. Sci. U.S.A.

131

188

View full text Add to dashboard Cite

Laboratory-based mock crime studies have often been interpreted to mean that (i) eyewitness confidence in an identification made from a lineup is a weak indicator of accuracy and (ii) sequential lineups are diagnostically superior to traditional simultaneous lineups. Largely as a result, juries are increasingly encouraged to disregard eyewitness confidence, and up to 30% of law enforcement agencies in the United States have adopted the sequential procedure. We conducted a field study of actual eyewitnesses who were assigned to simultaneous or sequential photo lineups in the Houston Police Department over a 1-y period. Identifications were made using a three-point confidence scale, and a signal detection model was used to analyze and interpret the results. Our findings suggest that (i) confidence in an eyewitness identification from a fair lineup is a highly reliable indicator of accuracy and (ii) if there is any difference in diagnostic accuracy between the two lineup formats, it likely favors the simultaneous procedure.eyewitness identification | confidence-accuracy relationship | simultaneous vs. sequential lineups E yewitnesses to a crime are often called upon by police investigators to identify a suspected perpetrator from a lineup. A traditional police lineup in the United States consists of the simultaneous presentation of six people, one of whom is the suspect (who is either guilty or innocent) and five of whom are fillers who resemble the suspect but who are known to be innocent. Live lineups were once the norm, but, nowadays, photo lineups are much more commonly used (1). When presented with a photo lineup, an eyewitness can identify someone-either the suspect (a suspect ID) or one of the fillers (a filler ID)-or can reject the lineup (no ID). A filler ID is a known error that does not imperil the identified individual, but a suspect ID (including a misidentification of an innocent suspect) does. According to the Innocence Project, eyewitness misidentification is the single greatest cause of wrongful convictions in the United States, having played a role in over 70% of the 333 wrongful convictions that have been overturned by DNA evidence since 1989 (2).In an effort to reduce eyewitness misidentifications, several reforms based largely on the results of mock crime studies have been proposed. In a typical mock crime study, participants become witnesses to a staged crime (e.g., a purse snatching) and then later attempt to identify the perpetrator from a targetpresent lineup (containing a photo of the perpetrator) or a target-absent lineup (in which the photo of the perpetrator is replaced by a photo of the "innocent suspect"). The results of mock crime studies have often been interpreted to mean that (i) eyewitness confidence is an unreliable indicator of accuracy (3, 4) and (ii) suspect ID accuracy is enhanced-and the risk to innocent suspects is reduced-when the lineup members are presented sequentially (i.e., one at a time) rather than simultaneously (5-7). In light of such findings, the state of New J...

show abstract

“…However, despite these apparent indicators of diagnostic superiority, an inquiry into the nature of ROC analysis, a wellestablished technique grounded in signal-detection theory (Green & Swets, 1966;Swets, Dawes, & Monahan, 2000), reveals that a higher diagnosticity ratio does not actually identify the superior procedure. Indeed, the field of medicine long ago abandoned the use of the diagnosticity ratio (where it is usually referred to as either the likelihood ratio or the positive likelihood ratio) and has come to instead rely almost exclusively on ROC analysis (Lusted, 1971a(Lusted, , 1971bMetz, 1978). We argue that a similar change in emphasis is needed in the field of eyewitness memory, and we begin our case by describing how ROC analysis is routinely used in the field of medicine to evaluate the performance of competing diagnostic tests.…”

mentioning

confidence: 99%

Receiver operating characteristic analysis of eyewitness memory: Comparing the diagnostic accuracy of simultaneous versus sequential lineups.

Mickes¹,

Flowe²,

Wixted³

2012

Journal of Experimental Psychology: Applied

190

349

View full text Add to dashboard Cite

A police lineup presents a real-world signal-detection problem because there are two possible states of the world (the suspect is either innocent or guilty), some degree of information about the true state of the world is available (the eyewitness has some degree of memory for the perpetrator), and a decision is made (identifying the suspect or not). A similar state of affairs applies to diagnostic tests in medicine because, in a patient, the disease is either present or absent, a diagnostic test yields some degree of information about the true state of affairs, and a decision is made about the presence or absence of the disease. In medicine, receiver operating characteristic (ROC) analysis is the standard method for assessing diagnostic accuracy. By contrast, in the eyewitness memory literature, this powerful technique has never been used. Instead, researchers have attempted to assess the diagnostic performance of different lineup procedures using methods that cannot identify the better procedure (e.g., by computing a diagnosticity ratio). Here, we describe the basics of ROC analysis, explaining why it is needed and showing how to use it to measure the performance of different lineup procedures. To illustrate the unique advantages of this technique, we also report 3 ROC experiments that were designed to investigate the diagnostic accuracy of simultaneous versus sequential lineups. According to our findings, the sequential procedure appears to be inferior to the simultaneous procedure in discriminating between the presence versus absence of a guilty suspect in a lineup.

show abstract

Signal Detectability and Medical Decision-Making

Cited by 278 publications

References 15 publications

Human vs. machine: evaluation of fluorescence micrographs

Human vs. machine: evaluation of fluorescence micrographs

Estimating the reliability of eyewitness identifications from police lineups

Receiver operating characteristic analysis of eyewitness memory: Comparing the diagnostic accuracy of simultaneous versus sequential lineups.

Contact Info

Product

Resources

About