Certainty about uncertainty in sleep staging: a theoretical framework

Abstract

Sleep stage classification is an important tool for the diagnosis of sleep disorders. Because sleep staging has such a high impact on clinical outcome, it is important that it is done reliably. However, it is known that uncertainty exists in both expert scorers and automated models. On average, agreement between human scorers is only 82.6%. In this manuscript, we provide a theoretical framework to facilitate discussion and further analyses of uncertainty in sleep staging. To this end, we introduce two variants of uncertainty, known from statistics and the machine learning community: aleatoric and epistemic uncertainty. We discuss what these types of uncertainties are, why the distinction is useful, where they arise from in sleep staging, and provide recommendations on how this framework can improve sleep staging in the future.

Interpretation and Further Development of the Hypnodensity Representation of Sleep Structure

“…Nevertheless, one should take into account that a supervised classifier only mimics the label distribution under the assumptions that the the model is evaluated on a test dataset that entails the same statistics as the training set, and that the model exhibits the 'right amount' of capacity. In other words, models that are evaluated on an out-of-distribution dataset, or models that are too small exhibit a large amount of epistemic uncertainty [7], which increases the average entropy of the hypnodensity graph. On the other hand, a model that has too much capacity has the tendency to over-fit on the training set and becomes over-confident, creating a lower-entropy hypnodensity graph.…”

“…This scoring ambiguity can be caused by inherently ambiguous epochs (see fig. 1), and the stochastic nature of human decision making [7]. To model this stochastic decision process, we model each expert annotation as a sample from a label distribution, being a probability distribution over sleep stages, that is conditioned upon the mixture coefficients of the characteristics belonging to these stages.…”

“…Nevertheless, the expert is forced to make a choice, which might give us the (possibly false) impression that the final choice is the 'ground-truth' sleep stage with 100% certainty, while in reality the same expert could possible have selected a different sleep stage when you had asked at another moment in time [9]. For the interested reader, a further discussion on sources that can be the underlying cause of this effect can be found in [7].…”

“…As also mentioned in section 6.2, the average entropy of predictions for all test set epoch was found to be H = 0.29 ± 0.04 for the supervised, and away by keeping an expert in the loop. The interested reader is referred to [7] for a further discussion on uncertainty in human vs model sleep scorers.…”

“…A PSG is scored epoch-based according to the AASM rules. These rules force the scorer to choose one sleep stage, but ambiguity remains within and across scorers due to epochs being ambiguous, or additional sources that lead to scoring ambiguity [7]. This ambiguity gives rise to a label distribution for each epoch.…”