Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials

Abstract

Background and Purpose. Assessment of the quality of randomized controlled trials (RCTs) is common practice in systematic reviews. However, the reliability of data obtained with most quality assessment scales has not been established. This report describes 2 studies designed to investigate the reliability of data obtained with the Physiotherapy Evidence Database (PEDro) scale developed to rate the quality of RCTs evaluating physical therapist interventions. Method. In the first study, 11 raters independently rated 25 RCTs randomly selected from the PEDro database. In the second study, 2 raters rated 120 RCTs randomly selected from the PEDro database, and disagreements were resolved by a third rater; this generated a set of individual rater and consensus ratings. The process was repeated by independent raters to create a second set of individual and consensus ratings. Reliability of ratings of PEDro scale items was calculated using multirater kappas, and reliability of the total (summed) score was calculated using intraclass correlation coefficients (ICC [1,1]). Results. The kappa value for each of the 11 items ranged from .36 to .80 for individual assessors and from .50 to .79 for consensus ratings generated by groups of 2 or 3 raters. The ICC for the total score was .56 (95% confidence interval=.47–.65) for ratings by individuals, and the ICC for consensus ratings was .68 (95% confidence interval=.57–.76). Discussion and Conclusion. The reliability of ratings of PEDro scale items varied from “fair” to “substantial,” and the reliability of the total PEDro score was “fair” to “good.”

Meta-Analysis of Acute Exercise Effects on State Anxiety: An Update of Randomized Controlled Trials Over the Past 25 Years

“…c Item 1 is not included in the calculation of the overall score as per instructions. [55] Such evidence supports statements, conclusions, and recommendations for the utilization of acute exercise for managing the daily manifestations of state anxiety in adults. The main analysis further indicated substantial heterogeneity in the overall ES.…”

“…The score of 6 has been previously determined as the cut-off point for a high quality study (i.e., Level I evidence). [55] Regarding baseline scores on the measures of anxiety symptoms, samples in 12 of the studies had scores above the threshold for average anxiety levels as reported/assessed by the authors. We were unable to make such determination of baseline anxiety levels if the study used an anxiety measure without established anxiety threshold cut-off scores (e.g., POMS, 10-item STAI, AD-ACL).…”

“…A cut-off score of 6 was selected for categorization of studies with regards to quality as this is a score previously established as a benchmark for a "good quality" RCT. [55] Those variables were selected a priori based on the moderators addressed by the previous meta-analyses on the topic [1] or new opportunities for moderators based on the focus of research over the past 25 years. The moderator of anxiety manipulation was selected in an attempt to compare the effects from studies using samples with above vs. within normal levels of anxiety.…”

Hip and Knee Strengthening Is More Effective Than Knee Strengthening Alone for Reducing Pain and Improving Activity in Individuals With Patellofemoral Pain: A Systematic Review With Meta-analysis

“…Reliability of the total score is 0.68 (95% CI: 0.57, 0.76) for consensus ratings. 26 When a trial was not included in the PEDro database, it was scored by a reviewer who had completed the PEDro scale training tutorial. Participants Studies had to include individuals with patellofemoral pain.…”

Effectiveness of functional training on cardiorespiratory parameters: a systematic review and meta‐analysis of randomized controlled trials

“…The methodological quality of the included studies was assessed by two independent investigators (MPCRB and LCMV) using the 0-10 PEDro scale (Maher et al, 2003;Macedo et al, 2010). For studies already assessed at PEDro database, we used their scores (http://www.pedro.…”