Generalization of the Theory of Signal Detectability ton-Eventm-Dimensional Forced-Choice Tasks

Scurfield, Brian K.

doi:10.1006/jmps.1997.1183

Cited by 38 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Connections between the two tasks were extended to the new measure of detectability, D SI , and the channel capacity, C 2I , of the 2IFC observer. These measures have been shown elsewhere to have a number of properties desirable in a measure of detectability (Scurfield, 1996(Scurfield, , 1998. Nonparametric bounds on A 2I in terms of A SI were derived, and similar bounds for D 2I in terms of D SI .…”

Section: Discussionmentioning

confidence: 93%

“…Finally, all these results are based on the relationship A SI =HR(0, 1 2 ) 2I . Elsewhere, Scurfield (1996Scurfield ( , 1998 has extended A SI =P(C) 2I to three events and to unidimensional and multidimensional decision axes by specifying an equality between one of the three-event, single-interval, three-alternative ROC volumes and the corresponding proportion of correct decisions in the three-event, three-interval, six-alternative task (for equal prior probabilities). Furthermore, he showed that D=C for the same tasks, respectively, and that results are generalizable to n-events.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Nonparametric Relationships between Single-Interval and Two-Interval Forced-Choice Tasks in the Theory of Signal Detectability

Miller¹,

Scurfield²,

Drga³

et al. 2002

Journal of Mathematical Psychology

Self Cite

View full text Add to dashboard Cite

Green's relationship, A SI =P(C) 2I , which equates the area, A SI , under the receiver operating characteristic (ROC) curve in the single-interval forcedchoice (SIFC) task with the proportion correct, P(C) 2I , in the two-interval forced-choice (2IFC) task, is rederived using the cross-correlation functions of the SIFC evidence distributions. The relationship is generalized to include discrete random variables, unidimensional decision axes that do not need to be monotonic with likelihood ratio, and arbitrary prior and guessing probabilities. A 2IFC difference decision rule is assumed. Further nonparametric relationships, including an equality between an entropy transform of A SI and the 2IFC channel capacity, nonparametric bounds on the area under the 2IFC ROC curve in terms of A SI , and methods for estimating 2IFC ROC curves based on information from the SIFC task, are developed. These relationships are investigated experimentally. Experiment I is a frequency-discrimination task where the evidence is known to be distributed as a discrete random variable. Experiment II is an amplitude-discrimination task where the theoretical evidence distributions are continuously distributed. The problem of observer inconsistency is addressed by repeating the experiments multiple times, using the same stimuli, then using group operating characteristic (GOC) analysis to remove unique noise. Results from Experiment I show excellent support for all the theoretical relationships, and results from Experiment II show partial support for the theoretical relationships.

show abstract

Section: Discussionmentioning

confidence: 93%

Section: Discussionmentioning

confidence: 99%

Nonparametric Relationships between Single-Interval and Two-Interval Forced-Choice Tasks in the Theory of Signal Detectability

Miller¹,

Scurfield²,

Drga³

et al. 2002

Journal of Mathematical Psychology

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, multiclass ROC analysis is a theoretical problem whose solution has been eluded the community ever since the introduction of the binary ROC in the 1950s [19], [20]. Much work has been devoted to understanding the nature of a multiclass classification problem, and many metrics have been proposed to assess the performance of a multiclass classification task [12], [13], [19]- [32].…”

Section: Discussion Current Status Of the Proposed Three-class Romentioning

confidence: 99%

“…Scurfield has previously investigated "n-event, m-dimensional" forced-choice tasks [12]. In that work, Scurfield first reformulated the two-class decision rules by introducing, for mathematical convenience, two dummy parameters which do not play a role in the observer's decision.…”

Section: Theorymentioning

confidence: 99%

The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS)

Frey²

2008

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Previously, we have proposed a method for three-class receiver operating characteristic (ROC) analysis based on decision theory. In this method, the volume under a three-class ROC surface (VUS) serves as a figure-of-merit (FOM) and measures three-class task performance. The proposed three-class ROC analysis method was demonstrated to be optimal under decision theory according to several decision criteria. Further, an optimal three-class linear observer was proposed to simultaneously maximize the signal-to-noise ratio (SNR) between the test statistics of each pair of the classes provided certain data linearity condition. Applicability of this three-class ROC analysis method would be further enhanced by the development of an intuitive meaning of the VUS and a more general method to calculate the VUS that provides an estimate of its standard error. In this paper, we investigated the general meaning and usage of VUS as a FOM for threeclass classification task performance. We showed that the VUS value, which is obtained from a rating procedure, equals the percent correct in a corresponding categorization procedure for continuous rating data. The significance of this relationship goes beyond providing another theoretical basis for three-class ROC analysis-it enables statistical analysis of the VUS value. Based on this relationship, we developed and tested algorithms for calculating the VUS and its variance. Finally, we reviewed the current status of the proposed three-class ROC analysis methodology, and concluded that it extends and unifies decision theoretic, linear discriminant analysis, and psychophysical foundations of binary ROC analysis in a three-class paradigm.

show abstract

“…The interest on three-class (or, more generally, multi-class) classification has prompted many researchers to investigate how ROC analysis can be generalized to multi-class problems [5][6][7][8][9][10][11][12][13]. Some of these investigations [7,[9][10][11][12][13] studied the three-class ROC problem when the decision variables are related to two likelihood ratios generated by the ideal observer, (see (3) below), while others studied the problem for one or two decision variables that may not necessarily represent the optimal decision variables [5,6,8].…”

Section: Introductionmentioning

confidence: 99%

Performance Analysis of Three-Class Classifiers: Properties of a 3-D ROC Surface and the Normalized Volume Under the Surface for the Ideal Observer

Sahiner

Chan

Hadjiiski

2008

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Classification of a given observation to one of three classes is an important task in many decision processes or pattern recognition applications. A general analysis of the performance of three-class classifiers results in a complex six-dimensional (6D) receiver operating characteristic (ROC) space, for which no simple analytical tool exists at present. We investigate the performance of an ideal observer under a specific set of assumptions that reduces the 6D ROC space to 3D by constraining the utilities of some of the decisions in the classification task. These assumptions lead to a 3D ROC space in which the true-positive fraction (TPF) can be expressed in terms of the two types of false-positive fractions (FPFs). We demonstrate that the TPF is uniquely determined by, and therefore is a function of, the two FPFs. The domain of this function is shown to be related to the decision boundaries in the likelihood ratio plane. Based on these properties of the 3D ROC space, we can define a summary measure, referred to as the normalized volume under the surface (NVUS), that is analogous to the area under the ROC curve (AUC) for a two-class classifier. We further investigate the properties of the 3D ROC surface and the NVUS for the ideal observer under the condition that the three class distributions are multivariate normal with equal covariance matrices. The probability density functions (pdfs) of the decision variables are shown to follow a bivariate log-normal distribution. By considering these pdfs, we express the TPF in terms of the FPFs, and integrate the TPF over its domain numerically to obtain the NVUS. In addition, we performed a Monte Carlo simulation study, in which the 3D ROC surface was generated by empirical "optimal" classification of case samples in the multi-dimensional feature space following the assumed distributions, to obtain an independent estimate of NVUS. The NVUS value obtained by using the analytical pdfs was found to be in good agreement with that obtained from the Monte Carlo simulation study. We also found that, under all conditions studied, the NVUS increased when the difficulty of the classification task was reduced by changing the parameters of the class distributions, thereby exhibiting the properties of a performance metric in analogous to AUC. Our results indicate that, under the conditions that lead to our 3D ROC analysis, the performance of a 3-class classifier may be analyzed by considering the ROC surface, and its accuracy characterized by the NVUS.

show abstract

Generalization of the Theory of Signal Detectability ton-Eventm-Dimensional Forced-Choice Tasks

Cited by 38 publications

References 27 publications

Nonparametric Relationships between Single-Interval and Two-Interval Forced-Choice Tasks in the Theory of Signal Detectability

Nonparametric Relationships between Single-Interval and Two-Interval Forced-Choice Tasks in the Theory of Signal Detectability

The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS)

Performance Analysis of Three-Class Classifiers: Properties of a 3-D ROC Surface and the Normalized Volume Under the Surface for the Ideal Observer

Contact Info

Product

Resources

About