Validation Framework for Sleep Stage Scoring in Wearable Sleep Trackers and Monitors with Polysomnography Ground Truth

Nguyen, Quyen N.T.; Le, Toan; Huynh, Quyen; Setty, Arveity R.; Vo, Toi Van; Le, Trung Q.

doi:10.3390/clockssleep3020017

Cited by 16 publications

(6 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various devices have been developed for predicting OSA, including wearables and nearables. Wearables, such as smartwatches (eg, Apple Watch Series 8, Galaxy Watch5, Google Pixel Watch, Fitbit Sense 2) and ring-shaped devices (eg, Oura Ring 3), have gained popularity in the market . These devices typically use sensors like photoplethysmography and accelerometers or gyroscopes to track heart rate variability and movement.…”

Section: Discussionmentioning

confidence: 99%

In-Home Smartphone-Based Prediction of Obstructive Sleep Apnea in Conjunction With Level 2 Home Polysomnography

Han,

Kim,

Rhee

et al. 2024

JAMA Otolaryngol Head Neck Surg

View full text Add to dashboard Cite

ImportanceConsumer-level sleep analysis technologies have the potential to revolutionize the screening for obstructive sleep apnea (OSA). However, assessment of OSA prediction models based on in-home recording data is usually performed concurrently with level 1 in-laboratory polysomnography (PSG). Establishing the predictability of OSA using sound data recorded from smartphones based on level 2 PSG at home is important.ObjectiveTo validate the performance of a prediction model for OSA using breathing sound recorded from smartphones in conjunction with level 2 PSG at home.Design, Setting, and ParticipantsThis diagnostic study followed a prospective design, involving participants who underwent unattended level 2 home PSG. Breathing sounds were recorded during sleep using 2 smartphones, one with an iOS operating system and the other with an Android operating system, simultaneously with home PSG in participants’ own home environment. Participants were 19 years and older, slept alone, and had either been diagnosed with OSA or had no previous diagnosis. The study was performed between February 2022 and February 2023.Main Outcomes and MeasuresSensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the predictive model based on the recorded breathing sounds.ResultsOf the 101 participants included during the study duration, the mean (SD) age was 48.3 (14.9) years, and 51 (50.5%) were female. For the iOS smartphone, the sensitivity values at apnea-hypopnea index (AHI) levels of 5, 15, and 30 per hour were 92.6%, 90.9%, and 93.3%, respectively, with specificities of 84.3%, 94.4%, and 94.4%, respectively. Similarly, for the Android smartphone, the sensitivity values at AHI levels of 5, 15, and 30 per hour were 92.2%, 90.0%, and 92.9%, respectively, with specificities of 84.0%, 94.4%, and 94.3%, respectively. The accuracy for the iOS smartphone was 88.6%, 93.3%, and 94.3%, respectively, and for the Android smartphone was 88.1%, 93.1%, and 94.1% at AHI levels of 5, 15, and 30 per hour, respectively.Conclusions and RelevanceThis diagnostic study demonstrated the feasibility of predicting OSA with a reasonable level of accuracy using breathing sounds obtained by smartphones during sleep at home.

show abstract

Section: Discussionmentioning

confidence: 99%

In-Home Smartphone-Based Prediction of Obstructive Sleep Apnea in Conjunction With Level 2 Home Polysomnography

Han,

Kim,

Rhee

et al. 2024

JAMA Otolaryngol Head Neck Surg

View full text Add to dashboard Cite

show abstract

“…This study classified CSTs into 3 types: wearables, nearables, and airables. Wearable devices or wearables, such as smartwatches and ring-shaped devices, are generally worn by users to track sleep using sensors like photoplethysmography sensors and accelerometers [6,[12][13][14][15][16]. Nearable devices or nearables, placed near the body without direct contact, have radar or mattress pads to detect subtle movements during sleep [9,17].…”

Section: Introductionmentioning

confidence: 99%

“…Given the surge of diverse CSTs, it is necessary to conduct comprehensive and objective evaluations of the performance of these CSTs available in the market [4,15,[18][19][20][21]. Some studies compared CSTs and alternative tools available for sleep analysis, such as electroencephalography headbands [4] or subjective sleep diaries [12] (without employing the gold standard polysomnography), which failed to validate the consistency between CSTs and polysomnography.…”

Section: Introductionmentioning

confidence: 99%

Accuracy of 11 Wearable, Nearable, and Airable Consumer Sleep Trackers: Prospective Multicenter Validation Study

Lee,

Cho,

Cha

et al. 2023

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

Background Consumer sleep trackers (CSTs) have gained significant popularity because they enable individuals to conveniently monitor and analyze their sleep. However, limited studies have comprehensively validated the performance of widely used CSTs. Our study therefore investigated popular CSTs based on various biosignals and algorithms by assessing the agreement with polysomnography. Objective This study aimed to validate the accuracy of various types of CSTs through a comparison with in-lab polysomnography. Additionally, by including widely used CSTs and conducting a multicenter study with a large sample size, this study seeks to provide comprehensive insights into the performance and applicability of these CSTs for sleep monitoring in a hospital environment. Methods The study analyzed 11 commercially available CSTs, including 5 wearables (Google Pixel Watch, Galaxy Watch 5, Fitbit Sense 2, Apple Watch 8, and Oura Ring 3), 3 nearables (Withings Sleep Tracking Mat, Google Nest Hub 2, and Amazon Halo Rise), and 3 airables (SleepRoutine, SleepScore, and Pillow). The 11 CSTs were divided into 2 groups, ensuring maximum inclusion while avoiding interference between the CSTs within each group. Each group (comprising 8 CSTs) was also compared via polysomnography. Results The study enrolled 75 participants from a tertiary hospital and a primary sleep-specialized clinic in Korea. Across the 2 centers, we collected a total of 3890 hours of sleep sessions based on 11 CSTs, along with 543 hours of polysomnography recordings. Each CST sleep recording covered an average of 353 hours. We analyzed a total of 349,114 epochs from the 11 CSTs compared with polysomnography, where epoch-by-epoch agreement in sleep stage classification showed substantial performance variation. More specifically, the highest macro F1 score was 0.69, while the lowest macro F1 score was 0.26. Various sleep trackers exhibited diverse performances across sleep stages, with SleepRoutine excelling in the wake and rapid eye movement stages, and wearables like Google Pixel Watch and Fitbit Sense 2 showing superiority in the deep stage. There was a distinct trend in sleep measure estimation according to the type of device. Wearables showed high proportional bias in sleep efficiency, while nearables exhibited high proportional bias in sleep latency. Subgroup analyses of sleep trackers revealed variations in macro F1 scores based on factors, such as BMI, sleep efficiency, and apnea-hypopnea index, while the differences between male and female subgroups were minimal. Conclusions Our study showed that among the 11 CSTs examined, specific CSTs showed substantial agreement with polysomnography, indicating their potential application in sleep monitoring, while other CSTs were partially consistent with polysomnography. This study offers insights into the strengths of CSTs within the 3 different classes for individuals interested in wellness who wish to understand and proactively manage their own sleep.

show abstract

“…For these reasons, standards for the validation 9,34 and implementation 10,35 of new technologies have been put forward for sleep predictions. Further, the concept of evaluating the performance of devices for sleep prediction has emerged as a strategy for addressing the limitations of more traditional validation approaches 10,36 . With the establishment of these standards, applications of novel sensors and scoring strategies have been called for 11 .…”

Section: Introductionmentioning

confidence: 99%

Performance of a Multisensor Smart Ring to Evaluate Sleep: In-Lab and Home-Based Evaluation Relative to Polysomnography and Actigraphy: Importance of Generalized Versus Personalized Scoring

Grandner

Bromberg

Morrell

et al. 2022

Preprint

View full text Add to dashboard Cite

Study Objectives: Wearable sleep technology has rapidly expanded across the consumer market due to advances in technology and increased interest in personalized sleep assessment to improve health and performance. In this study, we tested the performance of a novel device, alongside other commercial wearables, against in-lab and at-home polysomnography (PSG). Methods: 36 healthy adults were assessed across 77 nights while wearing the Happy Ring, as well as the Actiwatch, Fitbit, Whoop, and Oura Ring devices. Subjects participated in a single night of in-lab PSG and 2 nights of at-home PSG. The Happy Ring includes sensors for skin conductance, movement, heart rate, and skin temperature. Epoch-by-epoch analyses compared the wearable de-vices to both in-lab and at-home PSG. The Happy Ring utilized two machine-learning derived scor-ing algorithms: a generalized algorithm that applied broadly to all users, and a personalized algorithm that adapted to the data of individual subjects. Results: Compared to in-lab PSG, the generalized and personalized algorithms demonstrated good sensitivity (94% and 93%, respectively) and specificity (70% and 83%, respectively). The other wearable devices also demonstrated good sensitivity (89%-94%) but lower specificity (19%-54%), relative to the Happy Ring. Accuracy was 91% for generalized and 92% for personalized algorithms, compared to other devices that ranged from 84%-88%. The generalized algorithm demonstrated an accuracy of 67%, 85%, and 85% for light, deep, and REM sleep, respectively. The personalized algorithm was 81%, 95%, and 92% accurate for light, deep, and REM sleep, re-spectively. Conclusions: The Happy Ring performed well at home and in the lab, especially regarding sleep-wake detection. The personalized algorithm demonstrated improved detection accuracy over the generalized approach and other devices, suggesting that adaptable, dynamic algorithms can enhance sleep detection accuracy.

show abstract

Validation Framework for Sleep Stage Scoring in Wearable Sleep Trackers and Monitors with Polysomnography Ground Truth

Cited by 16 publications

References 31 publications

In-Home Smartphone-Based Prediction of Obstructive Sleep Apnea in Conjunction With Level 2 Home Polysomnography

In-Home Smartphone-Based Prediction of Obstructive Sleep Apnea in Conjunction With Level 2 Home Polysomnography

Accuracy of 11 Wearable, Nearable, and Airable Consumer Sleep Trackers: Prospective Multicenter Validation Study

Performance of a Multisensor Smart Ring to Evaluate Sleep: In-Lab and Home-Based Evaluation Relative to Polysomnography and Actigraphy: Importance of Generalized Versus Personalized Scoring

Contact Info

Product

Resources

About