Intraobserver and Interobserver Agreement of Structural and Functional Software Programs for Measuring Glaucoma Progression

Abstract

IMPORTANCEIt is important to evaluate intraobserver and interobserver agreement using visual field (VF) testing and optical coherence tomography (OCT) software in order to understand whether the use of this software is sufficient to detect glaucoma progression and to make decisions regarding its treatment.OBJECTIVE To evaluate agreement in VF and OCT software among 5 glaucoma specialists. DESIGN, SETTING AND PARTICIPANTSThe printout pages from VF progression software and OCT progression software from 100 patients were randomized, and the 5 glaucoma specialists subjectively and independently evaluated them for glaucoma. Each image was classified as having no progression, questionable progression, or progression. The principal investigator classified the patients previously as without variability (normal) or with high variability among tests (difficult). Using both software, the specialists also evaluated whether the glaucoma damage had progressed and if treatment change was needed. One month later, the same observers reevaluated the patients in a different order to determine intraobserver reproducibility. MAIN OUTCOMES AND MEASURESIntraobserver and interobserver agreement was estimated using κ statistics and Gwet second-order agreement coefficient. The agreement was compared with other factors. RESULTSOf the 100 observed patients, half were male and all were white; the mean (SD) age was 69.7 (14.1) years. Intraobserver agreement was substantial to almost perfect for VF software (overall κ [95% CI], 0.59 [0.46-0.72] to 0.87 [0.79-0.96]) and similar for OCT software (overall κ [95% CI], 0.59 [0.46-0.71] to 0.85 [0.76-0.94]). Interobserver agreement among the 5 glaucoma specialists with the VF progression software was moderate (κ, 0.48; 95% CI, 0.41-0.55) and similar to OCT progression software (κ, 0.52; 95% CI, 0.44-0.59). Interobserver agreement was substantial in images classified as having no progression but only fair in those classified as having questionable glaucoma progression or glaucoma progression. Interobserver agreement was fair regarding questions about glaucoma progression (κ, 0.39; 95% CI, 0.32-0.48) and consideration about treatment changes (κ, 0.39; 95% CI, 0.32-0.48). The factors associated with agreement were the glaucoma stage and case difficulty. CONCLUSIONS AND RELEVANCEThere was substantial intraobserver agreement but moderate interobserver agreement among glaucoma specialists using 2 glaucoma progression software packages. These data suggest that these glaucoma progression software packages are insufficient to obtain high interobserver agreement in both devices except in patients with no progression. The low agreement regarding progression or treatment changes suggests that both software programs used in isolation are insufficient for decision making.

Evaluation of RETICs Glaucoma Diagnostic Calculators in Preperimetric Glaucoma

“… 19 We reported previously that the interobserver agreement was higher in images or tests classified as not showing progression than in those classified as having questionable or definitive glaucoma progression. 20 Consequently, it is not easy to evaluate changes or progression by optic disc photography or other subjective methods. The guidelines of the World Glaucoma Association and the European Glaucoma Society advocate regular monitoring of both structural and functional changes, particularly in patients with early glaucomatous damage.…”

A CNN-aided method to predict glaucoma progression using DARC (Detection of Apoptosing Retinal Cells)

“…Objective assessment is increasingly recognised as being important in glaucoma, as there is variable agreement between clinicians, even with technological aids. Poor agreement has been shown with respect to defining progression in patients using visual fields, OCT and stereophotography [10][3][45] [11]. Indeed, for this study, we asked five masked senior glaucoma specialists (co-authors) to grade for progression of patients using their clinical judgement based on optic disc assessment, OCT, visual fields and management history; unfortunately, there was variable agreement between them although three of the more senior clinicians (OBS 1, 2 and 3, Supplementary Figure S4), did appear to agree more than the more junior experts (OBS 4 and 5).…”

“…However, several studies have demonstrated there is great variability amongst clinicians in agreement over progression using standard assessments including SAP, OCT and optic disc stereo photography. [10] [11][3] [12] However, clinical grading is regarded as the gold standard in real world practice. In deep learning datasets, manual grading of retinal images is regarded as the "ground truth" and is essential in order to train and test AI strategies in the automated detection of diseases such as glaucoma.…”

Functional assessment of glaucoma: Uncovering progression

“…Fair to moderate interobserver agreement persists, however, even when clinicians have the opportunity to review automated indicators of progression such as visual field indices, Bebie curves (cumulative defect curves), HFA overview printouts, Guided Progression Analysis printouts, STATPAC2, and PROGRESSOR. 89,90,114,133,178 For example, Lin and coworkers asked 8 glaucoma experts and 8 comprehensive ophthalmologists to review 40 visual field series and determine whether they were progressing or stable. 114 For glaucoma experts, the median k values for interobserver agreement were 0.47, 0.60, and 0.43 when HFA printouts, STATPAC2, and PROGRESSOR were used, respectively.…”