The consistency of panelists’ appropriateness ratings: do experts produce clinically logical scores for rectal cancer treatment?

Treatment of patients with metastatic renal cell cancer

“…It has been demonstrated that appropriateness guidelines developed using this method are reproducible, 10 are consistent clinically, 11 and are correlated with clinical outcomes. 12 Oncology applications have included breast cancer, 13 melanoma, 14 colorectal cancer, 9,15 and hematologic malignancies.…”

Appropriateness of Treatments for Patients With Degenerative Ataxias: Recommendations by a Panel of Experts

“…This method, developed in the mid-1980s by the RAND Corporation and the School of Medicine of the University of California at Los Angeles (UCLA), combines the best available scientific evidence with the practical experience from experts in the field to yield a statement regarding the appropriateness or inappropriateness of medical and surgical procedures. It has been verified that appropriateness guidelines developed by means of RAM are reproducible [8], clinically consistent [9], and correlated with clinical outcomes [10]. The RAM has different stages: 1) systematic review of the available scientific literature on the procedure to be rated; 2) development of a list of clinical scenarios which categorise patients likely to be encountered in clinical practice for the procedure in question in terms of their specific symptoms and signs, medical history and test results; 3) selection and setting of an expert panel; and 4) rating by the experts the benefit-to-harm ratio of the procedure for each clinical scenario, following a two-round modified Delphi technique [7].…”

“…Each panelist was provided with his or her first round own ratings together with the frequency distribution of all the experts' responses. For each clinical scenario-treatment combination, the median score was given, interpreted as appropriate [7][8][9], uncertain [4][5][6] or inappropriate [1][2][3]. In addition, there was an assessment of agreement/dispersion of views using the following definitions: (1) Agreement: no more than two panelists rate the indication outside the 3-point region [1-3, 4-6, 7-9] containing the median, (2) Disagreement: at least two panelists rated the indication in the 1 -3 region, and at least two rated it in the 7 -9 region, and (3) Indeterminate: covers all other eventualities.…”

Application of appropriate use criteria for stress myocardial perfusion imaging at two academic medical centers: Compliance and association with image findings

“…If true, these types of clinical tools follow, rather than lead practice. Potential biases of the AUC method include variability based upon composition of the expert panel (Coulter, Adams, & Shekelle, 1995) and misclassification bias found within clinical vignettes resulting in the propagation of clinical dogma (Hodgson et al, 2005). The method may in fact perpetuate and even legitimize established practice biases and norms. Referring providers at both institutions might have been closely following the published 2005 AUC algorithm.…”