Assessment of Automated Scoring of Polysomnographic Recordings in a Population with Suspected Sleep-disordered Breathing

Pittman, Stephen D.; Macdonald, Mary M.; Fogel, Robert; Malhotra, Atul; Todros, Koby; Levy, Baruch; Geva, Amir B.; White, David P.

doi:10.1093/sleep/27.7.1394

Cited by 83 publications

(118 citation statements)

References 0 publications

Supporting

Mentioning

110

Contrasting

Order By: Relevance

“…We and others [5][6][7][8]11,12 have found that disagreements occur primarily between wakefulness and sleep, between stages N1 and N2 and between stages N2 and N3 ( Table 1). A number of digital techniques are available that can reduce these disagreements: 1) We have shown previously 14 that most wake/sleep disagreements between scorers occur when ORP is between the clearly awake range (> 2.0) and the consolidated sleep range (< 1.0).…”

Section: Potential Approaches To Mitigate Inter-scorer Variabilitymentioning

confidence: 76%

“…[1][2][3][4][5][6][7][8][9][10][11][12] It not only affects the diagnosis and management of sleep disorders but also confounds interpretation of outcome studies. The reasons for discrepancies between scorers are not well understood, and their identification may provide an opportunity for solutions.…”

Section: Introductionmentioning

confidence: 99%

“…However, ensuring adequate training will by no means solve the problem since substantial inter-scorer variability remains between highly experienced technologists. [1][2][3][4][5][6][7][8][9][10][11][12][13] 2) Inattention by one or both scorers: Polysomnography (PSG) scoring can be monotonous, and errors related to boredom and inattention may be expected. It may be anticipated that differences related to inattention would be eliminated if the scorers were asked to edit their own scores.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Staging Sleep in Polysomnograms: Analysis of Inter-Scorer Variability

Younes

Raneri

Hanly

2016

Journal of Clinical Sleep Medicine

110

106

View full text Add to dashboard Cite

Study Objectives: To determine the reasons for inter-scorer variability in sleep staging of polysomnograms (PSGs). Methods: Fifty-six PSGs were scored (5-stage sleep scoring) by 2 experienced technologists, (first manual, M1). Months later, the technologists edited their own scoring (second manual, M2) based upon feedback from the investigators that highlighted differences between their scoring. The PSGs were then scored with an automatic system (Auto) and the technologists edited them, epoch-by-epoch (Edited-Auto). This resulted in 6 different manual scores for each PSG. Epochs were classified as scorer errors (one M1 score differed from the other 5 scores), scorer bias (all 3 scores of each technologist were similar, but differed from the other technologist) and equivocal (sleep scoring was inconsistent within and between technologists). Results: Percent agreement after M1 was 78.9% ± 9.0% and was unchanged after M2 (78.1% ± 9.7%) despite numerous edits (≈40/PSG) by the scorers. Agreement in Edited-Auto was higher (86.5% ± 6.4%, p < 1E−9). Scorer errors (< 2% of epochs) and scorer bias (3.5% ± 2.3% of epochs) together accounted for < 20% of M1 disagreements. A large number of epochs (92 ± 44/PSG) with scoring agreement in M1 were subsequently changed in M2 and/or EditedAuto. Equivocal epochs, which showed scoring inconsistency, accounted for 28% ± 12% of all epochs, and up to 76% of all epochs in individual patients. Disagreements were largely between awake/NREM, N1/N2, and N2/N3 sleep. Conclusion: Inter-scorer variability is largely due to epochs that are difficult to classify. Availability of digitally identified events (e.g., spindles) or calculated variables (e.g., depth of sleep, delta wave duration) during scoring may greatly reduce scoring variability.

show abstract

Section: Potential Approaches To Mitigate Inter-scorer Variabilitymentioning

confidence: 76%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Staging Sleep in Polysomnograms: Analysis of Inter-Scorer Variability

Younes

Raneri

Hanly

2016

Journal of Clinical Sleep Medicine

110

106

View full text Add to dashboard Cite

show abstract

“…5 Their analysis was based on 31 sleep studies scored by 2 independent sleep technologists. There was 93.5% agreement on OSA severity with all disagreements occurring in the mild and moderate categories.…”

Section: Rules Are Provided By the Aasm Manual For The Scoring Of Slementioning

confidence: 99%

The American Academy of Sleep Medicine Inter-scorer Reliability Program: Respiratory Events

Rosenberg

Hout

2014

Journal of Clinical Sleep Medicine

113

View full text Add to dashboard Cite

Study Objectives: The American Academy of Sleep Medicine (AASM) Inter-scorer Reliability program provides a unique opportunity to compare a large number of scorers with varied levels of experience to determine agreement in the scoring of respiratory events. The objective of this paper is to examine areas of disagreement to inform future revisions of the AASM Manual for the Scoring of Sleep and Associated Events. Methods: The sample included 15 monthly records, 200 epochs each. The number of scorers increased steadily during the period of data collection, reaching more than 3,600 scorers by the fi nal record. Scorers were asked to identify whether an obstructive, mixed, or central apnea; a hypopnea; or no event was seen in each of the 200 epochs. The "correct" respiratory event score was defi ned as the score endorsed by the most scorers. Percentage agreement with the majority score was determined for each epoch and the mean agreement determined. Results:The overall agreement for scoring of respiratory events was 93.9% (κ = 0.92). There was very high agreement on epochs without respiratory events (97.4%), and the majority score for most of the epochs (87.8%) was no event. For the 364 epochs scored as having a respiratory event, overall agreement that some type of respiratory event occurred was 88.4% (κ = 0.77). The agreement for epochs scored as obstructive apnea by the majority was 77.1% (κ = 0.71), and the most common disagreement was hypopnea rather than obstructive apnea (14.4%). The agreement for hypopnea was 65.4% (κ = 0.57), with 16.4% scoring no event and 14.8% scoring obstructive apnea. The agreement for central apnea was 52.4% (κ = 0.41). A single epoch was scored as a mixed apnea by a plurality of scorers. Conclusions:The study demonstrated excellent agreement among a large sample of scorers for epochs with no respiratory events. Agreement for some type of event was good, but disagreements in scoring of apnea vs. hypopnea and type of apnea were common. A limitation of the analysis is that most of the records had normal breathing. A review of controversial events yielded no consistent bias that might be resolved by a change of scoring rules. I dentifi cation of respiratory events during sleep is critical to the diagnosis of sleep related breathing disorders, especially obstructive sleep apnea.1 Rules are provided by the AASM Manual for the Scoring of Sleep and Associated Events 2 (the Manual) and require not only identifi cation of events but classifi cation of types. Apnea types include obstructive, mixed, and central events. In addition, apneas are distinguished from hypopneas, defi ned as a partial decrease of air fl ow as measured using a nasal pressure transducer.Studies have attempted to assess scoring reliability for respiratory events, often employing non-standard techniques. A widely cited study published before the Manual used data from home sleep testing relying on a single nasal thermistor. 3The study defi ned apneas as a 75% or greater reduction of fl ow amplitude and hypopneas as a 30% or ...

show abstract

“…The other one aiming at evaluating performances of automated methods [10][11][12], compared to visual analysis. A recent publication demonstrated that on a dataset of 70 recordings, an automated method did not differ more than visual analysis from a reference scoring [13].…”

mentioning

confidence: 99%

EOG-based auto-staging: less is more

Berthomier

Brandewinder

2015

Sleep Breath

View full text Add to dashboard Cite

This issue of Sleep and Breathing presents the validation results of a new automated wake/sleep staging method based on EOG activity, developed by Jussi Virkkala from the Finnish Institute of Occupational Health. Classically, the automated method is compared to visual analysis, on an epoch by epoch basis. It reaches a level of global concordance of 88 % with a Kappa of 0.57. In other words, on the 248,696 epochs of the validation dataset, 212,138 were scored correctly in wake/sleep, that is as the human expert did it, and on 36,558 epochs, the two scorings differ.This level is considered good in literature focusing on evaluating automated methods. It shows as follows:1. Automated analysis methods are continuously developing [1]. 2. Performances increase.Performance wise, two trends in literature coexist: one aiming at evaluating inter-expert agreement (the percentage of epochs of a recording or a set of recordings for which two human scorers give exactly the same score), when not intraexpert agreement (the percentage of epochs of a recording or a set of recordings for which a human scorer give the same score, when scoring data twice within a given period of time) [2][3][4][5][6][7][8][9]. The other one aiming at evaluating performances of automated methods [10][11][12], compared to visual analysis. A recent publication demonstrated that on a dataset of 70 recordings, an automated method did not differ more than visual analysis from a reference scoring [13]. In other words, automated analysis can reach accuracy comparable to visual analysis. These levels of performances are new. Let us remember what automated analysis looked like only a few years ago. There was some vicious circle: automated analysis was disregarded, thus attracted only little attention and effort, and was therefore doomed to be unsatisfactory as it obviously takes talent and time to learn a machine to mimic the extremely complex operations that an experienced scorer does when scoring sleep. The vicious circle seems to turn virtuous as automated analysis becomes a topic of interest where highprofile research teams get involved.Now that the accuracy of this method is established, let us consider how it works. Indeed, when visual analysis is standardized, automated methods are very diverse: many alternative approaches to PSG are explored.As stated in the AASM manual, conventional PSG, which is necessary even for the not so simple discrimination between wake and sleep, requires a minimum of seven channels. Here, the proposed montage is respiratory polygraphy+3 sensors (2 EOG+ref). The EOG-based method validated in this paper belongs to a set of methods which all tend to reduce the number of sensors on the patient: actimetry [14], peripheral arterial tone and pulse transit time [15], motion analysis [16], EOG [17], and EEG only [18][19][20][21][22][23][24]. One question immediately appears: could the discrepancies observed between visual staging and the validated methods be explained by this reduction in the number of signals? Probably not, as in thi...

show abstract

Assessment of Automated Scoring of Polysomnographic Recordings in a Population with Suspected Sleep-disordered Breathing

Cited by 83 publications

References 0 publications

Staging Sleep in Polysomnograms: Analysis of Inter-Scorer Variability

Staging Sleep in Polysomnograms: Analysis of Inter-Scorer Variability

The American Academy of Sleep Medicine Inter-scorer Reliability Program: Respiratory Events

EOG-based auto-staging: less is more

Contact Info

Product

Resources

About