Assessing cognitive screeners with the critical success index

Larner, AJ

doi:10.1002/pnp.719

Cited by 20 publications

(20 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Addressing the class imbalance problem using methods which oversample the minority class (in the current example TP cases), such as variants of the synthetic majority oversampling technique, SMOTE [21], or which undersample the majority class (in the current example TN) might be applicable. Comparisons of EI with measures such as the F measure [2,20] or the critical success index [2,24] which eschew TN values might be particularly pertinent in this class imbalance situation.…”

Section: Limitationsmentioning

confidence: 99%

Efficiency Index for Binary Classifiers: Concept, Extension, and Application

Larner¹

2023

Preprint

View full text Add to dashboard Cite

Many metrics exist for the evaluation of binary classifiers, all with their particular advantages and shortcomings. Recently an “Efficiency Index” for the evaluation of classifiers has been proposed, based on the consistency, or matching, and contradiction, or mismatching, of outcomes. This metric and its confidence intervals are easy to calculate from base data in a 2x2 contingency table, and values can be qualitatively and semi-quantitatively categorised. For medical tests, in which context the Efficiency Index was originally proposed, it facilitates communication of risk (of correct diagnosis versus misdiagnosis) to both clinicians and patients. Variants of the efficiency index (balanced, unbiased) which take into account disease prevalence and test cut-offs have also been described. The objectives of the current paper were firstly to extend the EI construct to other formulations (balanced level, quality) and secondly to explore the utility of EI and all four of its variants when applied to the dataset of a large prospective test accuracy study of a cognitive screening instrument. This showed that the balanced level, quality, and unbiased formulations of EI are more stringent measures.

show abstract

Section: Limitationsmentioning

confidence: 99%

Efficiency Index for Binary Classifiers: Concept, Extension, and Application

Larner¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Moreover, the CSI detection performance does not change as a function of event frequency [ 34 ], making it suitable for diverse count-rate scenarios. Although CSI is widely used to forecast weather events, it has been applied in other disciplines, when discrimination of rare events is required [ 35 ].…”

Section: Simplified Correlation Indexmentioning

confidence: 99%

A Simplified Correlation Index for Fast Real-Time Pulse Shape Recognition

Cicuttin

Morales

Crespo

et al. 2022

Sensors

View full text Add to dashboard Cite

A simplified correlation index is proposed to be used in real-time pulse shape recognition systems. This index is similar to the classic Pearson’s correlation coefficient, but it can be efficiently implemented in FPGA devices with far fewer logic resources and excellent performance. Numerical simulations with synthetic data and comparisons with the Pearson’s correlation show the suitability of the proposed index in applications such as the discrimination and counting of pulses with a predefined shape. Superior performance is evident in signal-to-noise ratio scenarios close to unity. FPGA implementation of Person’s method and the proposed correlation index have been successfully tested and the main results are summarized.

show abstract

“…In signal detection theory, CSI is defined as the ratio of hits to the sum of hits, false alarms, and misses. 8,9 CSI may also be expressed in terms of Sens and PPV 7 : CSI values range from 0 to 1, interpreted as 0 = unable to forecast and 1 = perfect forecast. 5,10 CSI is not unbiased, because CSI = TP/(sample size N -TN), giving lower scores for rarer events.…”

mentioning

confidence: 99%

“…5,10 CSI is not unbiased, because CSI = TP/(sample size N -TN), giving lower scores for rarer events. 9 In such circumstances, GS (also known as equitable threat score 5,11 ) may be preferred, as it takes into account the number of hits due to chance (CH), 8 where: or 9 :…”

mentioning

confidence: 99%

Using Critical Success Index or Gilbert Skill Score as composite measures of positive predictive value and sensitivity in diagnostic accuracy studies: Weather forecasting informing epilepsy research

et al. 2023

View full text Add to dashboard Cite

The Critical Success Index (CSI) and Gilbert Skill score (GS) are verification measures that are commonly used to check the accuracy of weather forecasting.In this article, we propose that they can also be used to simplify the joint interpretation of positive predictive value (PPV) and sensitivity estimates across diagnostic accuracy studies of epilepsy data. This is because CSI and GS each provide a single measure that takes the weather forecasting equivalent of PPV and sensitivity into account. We have re-analysed data from our recent systematic review of diagnostic accuracy studies of administrative epilepsy data using CSI and GS. We summarise the results and benefits of this approach.

show abstract

Assessing cognitive screeners with the critical success index

Cited by 20 publications

References 27 publications

Efficiency Index for Binary Classifiers: Concept, Extension, and Application

Efficiency Index for Binary Classifiers: Concept, Extension, and Application

A Simplified Correlation Index for Fast Real-Time Pulse Shape Recognition

Using Critical Success Index or Gilbert Skill Score as composite measures of positive predictive value and sensitivity in diagnostic accuracy studies: Weather forecasting informing epilepsy research

Contact Info

Product

Resources

About