Background and Purpose-Angiographic occlusion as a surrogate marker of satisfactory aneurysm treatment is commonly used in clinical trials although some pitfalls have to be considered. To investigate the inter-rater reliability of visual rating of aneurysm occlusion as study end point, we performed a systematic review and meta-analysis. Methods-Electronic databases (MEDLINE, EMBASE, PubMed, and the Cochrane Library) were searched up to June 2014. Assessment of risk for bias was based on the Quality Appraisal Tool for Studies of Diagnostic Reliability and the Guidelines for Reporting Reliability and Agreement studies. Inter-rater reliability estimates were pooled across studies using meta-analysis, and the influence of several factors (eg, imaging methods, grading scales, and occlusion rate) was tested with meta-regression. Results-From 1193 titles, 644 abstracts and 87 full-text versions were reviewed. Twenty-six articles met the inclusion criteria and provided 77 reliability estimates. Twenty-one different rating scales were used, and statistical analysis varied. Mean inter-rater agreement of the pooled studies was substantial (κ=0.65; 95% confidence interval, 0.60-0.69).Reliability varied significantly as a function of imaging methods, grading scales, occlusion rates, and their interaction. Observer agreement substantially increased with increasing occlusion rate in digital subtraction angiography but not in MR angiography. Reliability was higher in studies using 2-or 3-value grading scales than in studies with 4-value grading scales. Conclusions-There is significant heterogeneity between studies evaluating the reliability of visual evaluation of aneurysm coiling. On the basis of our analysis, we found that the combination of magnetic resonance angiography, 3-value grading scale, and 2 trained raters seems most promising for usage as surrogate study end points.
Ernst et al
Reliability of Visual Evaluation of Aneurysm Coiling 1575
Data Sources and Study SelectionStudies were identified by searching electronic databases (Ovid MEDLINE and EMBASE, PubMed, and the Cochrane Library) up to June 2014. The full electronic search strategy for all databases is presented in Table I in the online-only Data Supplement. In addition, we handsearched systematic reviews and high-impact journals in the field of interventional neuroradiology (Stroke, American Journal of Neuroradiology, and Neuroradiology).The reference lists of all retrieved articles were checked for other relevant articles. Duplicates were removed, and the titles and abstracts of the retrieved articles were reviewed for appropriateness to the study question. Case reports, conference abstracts, reviews, editorials, meta-analysis, guidelines, and animal studies were excluded. Only articles in English, German, French, and Spanish were screened.From the remaining articles, full-text versions were reviewed in detail. Studies were eligible for inclusion if they evaluated the reliability of MRA or DSA of patients with intracranial aneurysms treated with endovascular c...