The American Academy of Sleep Medicine Inter-scorer Reliability Program: Respiratory Events

Abstract

Study Objectives: The American Academy of Sleep Medicine (AASM) Inter-scorer Reliability program provides a unique opportunity to compare a large number of scorers with varied levels of experience to determine agreement in the scoring of respiratory events. The objective of this paper is to examine areas of disagreement to inform future revisions of the AASM Manual for the Scoring of Sleep and Associated Events. Methods: The sample included 15 monthly records, 200 epochs each. The number of scorers increased steadily during the period of data collection, reaching more than 3,600 scorers by the fi nal record. Scorers were asked to identify whether an obstructive, mixed, or central apnea; a hypopnea; or no event was seen in each of the 200 epochs. The "correct" respiratory event score was defi ned as the score endorsed by the most scorers. Percentage agreement with the majority score was determined for each epoch and the mean agreement determined. Results:The overall agreement for scoring of respiratory events was 93.9% (κ = 0.92). There was very high agreement on epochs without respiratory events (97.4%), and the majority score for most of the epochs (87.8%) was no event. For the 364 epochs scored as having a respiratory event, overall agreement that some type of respiratory event occurred was 88.4% (κ = 0.77). The agreement for epochs scored as obstructive apnea by the majority was 77.1% (κ = 0.71), and the most common disagreement was hypopnea rather than obstructive apnea (14.4%). The agreement for hypopnea was 65.4% (κ = 0.57), with 16.4% scoring no event and 14.8% scoring obstructive apnea. The agreement for central apnea was 52.4% (κ = 0.41). A single epoch was scored as a mixed apnea by a plurality of scorers. Conclusions:The study demonstrated excellent agreement among a large sample of scorers for epochs with no respiratory events. Agreement for some type of event was good, but disagreements in scoring of apnea vs. hypopnea and type of apnea were common. A limitation of the analysis is that most of the records had normal breathing. A review of controversial events yielded no consistent bias that might be resolved by a change of scoring rules. I dentifi cation of respiratory events during sleep is critical to the diagnosis of sleep related breathing disorders, especially obstructive sleep apnea.1 Rules are provided by the AASM Manual for the Scoring of Sleep and Associated Events 2 (the Manual) and require not only identifi cation of events but classifi cation of types. Apnea types include obstructive, mixed, and central events. In addition, apneas are distinguished from hypopneas, defi ned as a partial decrease of air fl ow as measured using a nasal pressure transducer.Studies have attempted to assess scoring reliability for respiratory events, often employing non-standard techniques. A widely cited study published before the Manual used data from home sleep testing relying on a single nasal thermistor. 3The study defi ned apneas as a 75% or greater reduction of fl ow amplitude and hypopneas as a 30% or ...

AI vs Humans for the diagnosis of sleep apnea

“…Expert scoring shows similar level of performance across each sleep expert. However, the expert agreement was relatively low when scoring individual apnea-hypopnea events, consistent with previous results [12]. Performance at the Diagnosis scale: To evaluate performance at the record scale, two metrics were used: Mean AHI Error and Diagnostic Accuracy of OSA severity.…”

“…The AHI allows for the evaluation of the severity of OSA, ranging from mild (5 < AHI < 15) to severe (AHI > 30). However, the process of scoring individual breathing events is tedious, costly, and subject to variability between trained experts [12].…”

Validation of a Suprasternal Pressure Sensor for Sleep Apnea Classification in Children

“…As the PneaVox sensor used in the present study is similar to the sensor used in comparison to the esophageal pressure in adult patients, in the study by Meslier et al, 8 we may reasonably assume that the agreement between the SSP and the esophageal pressure is similar in children. The difference in the number of events scored by the two readers may be explained by an inter-scorer variability, as recently assessed in large study by Rosenberg et al 13 As the PneaVox sensor has not been validated for detecting hypopneas; therefore, we only included apneic events in our analysis. 8 In conclusion, the SSP of the PneaVoX sensor is useful for the classification of apneas in children.…”

The Dreem Headband as an Alternative to Polysomnography for EEG Signal Acquisition and Sleep Staging

“…more than 3,200,000 scoring decisions). The agreement was highest for the REM sleep stage (90.5%) and slightly lower for N2 and Wake (85.2% and 84.1%, respectively), while the agreement was far lower for stages N3 and N1 (67.4% and 63.0%, respectively), placing constraints on the reliability of manual scoring (7). Critically, studies also indicate that agreement varies substantially across different sleep pathologies and sleep centers (7,8).…”