The importance of discriminative power rather than significance when evaluating potential clinical biomarkers in epilepsy research

Abstract

Objective:The quest for epilepsy biomarkers is on the rise. Variables with statistically significant group-level differences are often misinterpreted as biomarkers with sufficient discriminative power. This study aimed to demonstrate the relationship between significant group-level differences and a variable's power to discriminate between individuals. Methods:We simulated normal-distributed datasets from hypothetical populations with varying sample sizes (25-800), effect sizes (Cohen's d: .25-2.50), and variability (standard deviation: 10-35) to assess the impact of these parameters on significance and discriminative power. The simulation data were illustrated by assessing the discriminative power of a potential real-case biomarker-the EEG beta band power-to diagnose generalized epilepsy, using data from 66 children with generalized epilepsy and 385 controls. Additionally, we evaluated recently reported epilepsy biomarkers by comparing their effect sizes to our simulation-derived effect size criterion.Results: Group size affects significance but not discriminative power.Discriminative power is much more related to variability and effect size. Our real data example supported these simulation results by demonstrating that group-level significance does not translate, one to one, into discriminative power. Although we found a significant difference in the beta band power between children with and without epilepsy, the discriminative power was poor due to a small effect size. A Cohen's d of at least 1.25 is required to reach good discriminative power in univariable prediction modeling. Slightly over 60% of the biomarkers in our literature search met this criterion.Significance: Rather than statistical significance of group-level differences, effect size should be used as an indicator of a variable's biomarker potential. The minimal required effects size for individual biomarkers-a Cohen's d of 1.25-is large.This calls for multivariable approaches, in which combining multiple variables with smaller effect sizes could increase the overall effect size and discriminative power.

Eye-Tracking Biomarkers and Autism Diagnosis in Primary Care

“…These include less efficient nonsocial attentional disengagement, 33 , 34 , 35 amplitude and latency differences of the pupillary light reflex (PLR), 36 , 37 larger tonic baseline 38 , 39 or resting 40 pupil diameter, and differences in basic oculomotor measures, such as fixation duration 41 and saccade amplitude. 42 Although these are almost exclusively laboratory-based studies with effect sizes that suggest limited diagnostic utility, 43 these results indicate that multiple eye-tracking indices may be combined to increase discriminative power 44 and have promise for identifying autism in toddlers and young children.…”