Benchmark on a large cohort for sleep-wake classification with machine learning techniques

Abstract

Accurately measuring sleep and its quality with polysomnography (PSG) is an expensive task. Actigraphy, an alternative, has been proven cheap and relatively accurate. However, the largest experiments conducted to date, have had only hundreds of participants. In this work, we processed the data of the recently published Multi-Ethnic Study of Atherosclerosis (MESA) Sleep study to have both PSG and actigraphy data synchronized. We propose the adoption of this publicly available large dataset, which is at least one order of magnitude larger than any other dataset, to systematically compare existing methods for the detection of sleep-wake stages, thus fostering the creation of new algorithms. We also implemented and compared state-of-the-art methods to score sleep-wake stages, which range from the widely used traditional algorithms to recent machine learning approaches. We identified among the traditional algorithms, two approaches that perform better than the algorithm implemented by the actigraphy device used in the MESA Sleep experiments. The performance, in regards to accuracy and F 1 score of the machine learning algorithms, was also superior to the device’s native algorithm and comparable to human annotation. Future research in developing new sleep-wake scoring algorithms, in particular, machine learning approaches, will be highly facilitated by the cohort used here. We exemplify this potential by showing that two particular deep-learning architectures, CNN and LSTM, among the many recently created, can achieve accuracy scores significantly higher than other methods for the same tasks.

The future of sleep health: a data-driven revolution in sleep science and medicine

“…Penzel et al provided an in depth review of some of these approaches in clinical settings, offering a quantitative analysis of their performance and requirements 106 . Palotti et al evaluated the performance of some of the most common approaches, including statistical ML, on actigraphy data 107 .…”

“…Machine learning and deep learning approaches have gained traction in recent years for the task of classifying sleep-wake cycles and sleep stages in multi-modal sensor data 108,109 . With the availability of raw actigraphy signals, several deep-learning techniques such as convolutional neural networks 110 and recurrent neural networks 111 have been used to exploit the temporal nature of this unstructured data to distinguish the sleep-wake cycles 107 robustly and understand the role of activity in sleeprelated disorders 112 . Whilst the evaluation of most traditional and ML algorithms are performed using standard quality metrics such as accuracy, precision and recall per class, it is also important to measure clinically relevant metrics such as waking after sleep onset (WASO) and sleep efficiency 107 .…”

“…It is important to note that different methods are ought to be used based on the objective at hand. For instance, deep-learning methods often provide better performance than traditional statistical learning methods, but require large computational power and lack the interpretability that other models offer 107 . Performance and model evaluation is discussed in further detail on the Supplementary Material.…”

“…Recently, sleep data repositories, such as Sleepdata.org, have been created to advance the field 150 . These repositories include multimodal sleep data (from clinical-grade PSG to actigraphy and questionnaires) 150 , and are being used to create ML benchmarks 107 . These developments are crucial for the creation of generalised ML models that can be applied reliably to clinical and commercial settings to further our understanding of the role of sleep in well-being and disease.…”

Data-Driven Drug Repurposing for COVID-19

“…It has been shown that Random Forests, XGBoost and SVMs can be used efficiently for a variety of bioinformatics problems (Mall et al, 2017;Rawi et al, 2018;Mall et al, 2018;Ullah et al, 2018;Palotti et al, 2019;Elbasir et al, 2020).…”