EEG interpretation reliability and interpreter confidence: A large single-center study

Abstract

The intrarater and interrater reliability (I&IR) of EEG interpretation has significant implications for the value of EEG as a diagnostic tool. We measured both I&IR of EEG interpretation based on interpretation of complete EEGs into standard diagnostic categories and rater confidence in their interpretations, and investigated sources of variance in EEG interpretations. During two distinct time intervals six board-certified clinical neurophysiologists classified 300 EEGs into one or more of seven diagnostic categories, and assigned a subjective confidence to their interpretations. Each EEG was read by three readers. Each reader interpreted 150 unique studies, and 50 studies twice to generate intrarater data. A generalizability study assessed the contribution of subjects, readers, and the interaction between subjects and readers to interpretation variance. Five of the six readers had a median confidence of ≥ 99%, and the upper quartile of confidence values was 100% for all six readers. Intrarater Cohen’s kappa (κc) ranged from 0.33 to 0.73 with an aggregated value of 0.59. κc ranged from 0.29 to 0.62 for the 15 reader pairs, with an aggregated Fleiss kappa of 0.44 for interrater agreement. The κc were not significantly different across rater pairs (Chi-Square = 17.3, df=14, p = 0.24). Variance due to subjects (i.e. EEGs) was 65.3%, to readers was 3.9%, and to the interaction between readers and subjects was 30.8%. Experienced epileptologists have very high confidence in their EEG interpretations and low to moderate I&IR, a common paradox in clinical medicine. A necessary but insufficient condition to improve EEG interpretation accuracy is to increase intrarater and interrater reliability. This goal could be accomplished, for instance, with an automated on-line application integrated into a continuing medical education module that measures and reports EEG I&IR to individual users.

Hemodynamic correlates of spontaneous neural activity measured by human whole-head resting state EEG + fNIRS

“…However it is well known that EEG interpretation contains a substantial intuitive component and the accuracy of EEG interpretation is demonstrably low (Grant et al, 2014). These may well be due to our incomplete knowledge of its underlying mechanisms.…”

Diagnostic accuracy of microEEG: A miniature, wireless EEG device

“…Aware of this limitation, we performed a separate study of EEG intra- and inter-rater reliability [3]. Briefly, a pool of six epileptologists interpreted 300 EEGs in such a way as to generate both intra- and inter-rater reliability data, as well other variables of interest.…”

“…Third, estimation of the new device's true accuracy must account for the accuracy of both the reference device and the EEG interpreters. These challenges led to two appurtenant studies, the results of which are utilized in the present analyses [3, 4]. …”

Assessment of Olfactory Function in MAPT-Associated Neurodegenerative Disease Reveals Odor-Identification Irreproducibility as a Non-Disease-Specific, General Characteristic of Olfactory Dysfunction

“…Cohen’s kappa (κ), an index that measures inter-rater agreement for categorical items and takes into account agreement occurring by chance, was used to assess variability in an individual subject’s olfactory performance on successive UPSITs or BSITs. This measure, which is frequently used to evaluate agreement between different individuals, has also been used to evaluate inter-rater reliability (e.g., [43]) and is useful here since replicate tests of olfactory function made at six month or longer intervals should not be influenced by the memory of prior testing and can therefore be treated as independent assessments. Therefore, κ was used to evaluate how reproducibly a subject identified or misidentified individual odors in successive UPSIT and BSIT assessments.…”