IntelliSleepScorer, a software package with a graphic user interface for automated sleep stage scoring in mice based on a light gradient boosting machine algorithm

Abstract

Machine learning has been applied in recent years to categorize sleep stages (NREM, REM, and wake) using electroencephalogram (EEG) recordings; however, a well-validated sleep scoring automatic pipeline in rodent research is still not publicly available. Here, we present IntelliSleepScorer, a software package with a graphic user interface to score sleep stages automatically in mice. IntelliSleepScorer uses the light gradient boosting machine (LightGBM) to score sleep stages for each epoch of recordings. We developed LightGBM models using a large cohort of data, which consisted of 5776 h of sleep EEG and electromyogram (EMG) signals across 519 unique recordings from 124 mice. The LightGBM model achieved an overall accuracy of 95.2% and a Cohen’s kappa of 0.91, which outperforms the baseline models such as the logistic regression model (accuracy = 93.3%, kappa = 0.88) and the random forest model (accuracy = 94.3%, kappa = 0.89). The overall performance of the LightGBM model as well as the performance across different sleep stages are on par with that of the human experts. Most importantly, we validated the generalizability of the LightGBM models: (1) The LightGBM model performed well on two publicly available, independent datasets (kappa > = 0.80), which have different sampling frequency and epoch lengths; (2) The LightGBM model performed well on data recorded at a lower sampling frequency (kappa = 0.90); (3) The performance of the LightGBM model is not affected by the light/dark cycle; and (4) A modified LightGBM model performed well on data containing only one EEG and one EMG electrode (kappa > = 0.89). Taken together, the LightGBM models offer state-of-the-art performance for automatic sleep stage scoring in mice. Last, we implemented the IntelliSleepScorer software package based on the validated model to provide an out-of-box solution to sleep researchers (available for download at https://sites.broadinstitute.org/pan-lab/resources).

Sleep-Deep-Net learns sleep wake scoring from the end-user and completes each record in their style

“…In other words, while the average reliability of a data set may be arguably acceptable, there are typically some files that yield far poorer REM reliability. 15 Other methods do not have a means of flagging these unreliable scores.…”

“…Additionally, we report considerably higher overall F1 scores – 0.82 by WaveSleepNet vs 0.96 by SDN. IntelliSleepScorer 15 is another example of the more commonly found model that is trained on existing mouse EEG, expected to perform well on new data. IntelliSleepScorer provides 10 second epochs only whereas SDN does not require a predetermined epoch duration.…”

“…One crucial consideration with most models trained with existing mouse EEG and scores is the question of whether they will perform as well as in their source publication, when presented with novel data. 17 Both the WaveSleepNet 12 and IntelliSleepScorer 15 papers address this by obtaining outside data sets. In both papers, their models perform worse with outside data than with their in-house data.…”

“…In other words, while the average reliability of a data set may be arguably acceptable, there are typically some files that yield far poorer REM reliability. 15 Other methods do not have a means of flagging these unreliable scores. Investigators using other methods would need to accept that an unknown subset of the files in their data set were unreliable.…”

“…These models are then required to score new data based on what it has learned from the data with which they are familiar. [9][10][11][12][13][14][15][16] However, one concern with this approach is that networks can often perform poorly when they are asked to classify new data that they have not encountered before. 17 This could be the case with data from a different laboratory or unfamiliar experimental treatments, such as alterations in EEG/EMG produced by pharmacological agents, genetic manipulations, and/or disease models.…”