Exploring scoring methods for research studies: Accuracy and variability of visual and automated sleep scoring

Abstract

Sleep studies face new challenges in terms of data, objectives and metrics. This requires reappraising the adequacy of existing analysis methods, including scoring methods. Visual and automatic sleep scoring of healthy individuals were compared in terms of reliability (i.e., accuracy and stability) to find a scoring method capable of giving access to the actual data variability without adding exogenous variability. A first dataset (DS1, four recordings) scored by six experts plus an autoscoring algorithm was used to characterize inter‐scoring variability. A second dataset (DS2, 88 recordings) scored a few weeks later was used to explore intra‐expert variability. Percentage agreements and Conger's kappa were derived from epoch‐by‐epoch comparisons on pairwise and consensus scorings. On DS1 the number of epochs of agreement decreased when the number of experts increased, ranging from 86% (pairwise) to 69% (all experts). Adding autoscoring to visual scorings changed the kappa value from 0.81 to 0.79. Agreement between expert consensus and autoscoring was 93%. On DS2 the hypothesis of intra‐expert variability was supported by a systematic decrease in kappa scores between autoscoring used as reference and each single expert between datasets (.75–.70). Although visual scoring induces inter‐ and intra‐expert variability, autoscoring methods can cope with intra‐scorer variability, making them a sensible option to reduce exogenous variability and give access to the endogenous variability in the data.

Regular Caffeine Intake Delays REM Sleep Promotion and Attenuates Sleep Quality in Healthy Men

“…Sleep staging was performed with an automatic scoring algorithm (ASEEGA, version 4.4.23, PHYSIP, Paris, France) which has been successfully used in previous studies (Reichert et al, 2017;Gaggioni et al, 2019) and has been shown to reach good agreement with manual sleep scoring (Berthomier et al, 2020). Three data sets were excluded from analyses of allnight variables concerning the entire sleep episode due to technical issues (placebo condition: n = 2; caffeine condition: n = 1) and one volunteer was excluded from all analyses of REM sleep parameters (placebo condition) based on potential missed REM sleep episode by the algorithm and disagreement with visual scorers.…”

Alzheimer’s disease genetic risk and sleep phenotypes in healthy young men: association with more slow waves and daytime sleepiness

“…Scoring of sleep stages was performed automatically in 30-s epochs using a validated algorithm (ASEEGA, PHYSIP, Paris, France) 31 and according to 2017 American Academy of Sleep Medicine criteria, version 2.4. An automatic artefact detection algorithm with adapting thresholds 32 was further applied on scored data.…”

Alzheimer’s disease genetic risk and sleep phenotypes: association with more slow-waves and daytime sleepiness

“…EEG data were re-referenced off-line to average mastoids. Scoring of sleep stages was performed automatically in 30-s epochs using a validated algorithm (ASEEGA, PHYSIP, Paris, France) 31 and according to 2017 American Academy of Sleep Medicine criteria, version 2.4. An automatic artefact detection algorithm with adapting thresholds 32 was further applied on scored data.…”