Validation of a Global Rating Scale for Shoulder Arthroscopy: A Pilot Study

Abstract

Background Reliable, valid, feasible and objective assessment is desirable in all aspects of medical training to allow monitoring of progress and to ensure high standards. At present, no assessment model for shoulder arthroscopy exists that indisputably meets these criteria. The global rating scale (GRS) has been validated as a measure of technical performance in various surgical procedures. We have developed and aim to validate a GRS for shoulder arthroscopy (GRSSA). Materials and methods Using the GRSSA, 13 shoulder surgeons, rated 10 videos of diagnostic shoulder arthroscopy showing different levels of proficiency. Assessments were performed remotely using an online system developed for the study; assessors were blinded to the experience of the surgeon who was performing the arthroscopy. Construct validity was assessed via Cronbach's α and inter‐rater reliability was assessed via two‐way analysis of variance (ANOVA). Results The results suggest good construct validity (high internal consistency of GRSSA categories: Cronbach's α = 0.89 to 0.96). Two‐way ANOVA demonstrated a good discriminatory ability of the GRSSA but variability between raters (p = 0.0002 to 0.01), suggesting a current lack of inter‐rater reliability. Discussion The GRSSA appears to be a useful tool for the assessment of diagnostic arthroscopic skills and, using our online system, the assessor can be remote to the surgeon who is performing the procedure.

The Arthroscopic Surgical Skill Evaluation Tool (ASSET)

“…This assessment tool demonstrated construct validity, but their use of a single, non-blind observer did not allow for reliability testing 21 . In a similar process, Hoyle et al created the Global Rating Scale for Shoulder Arthroscopy (GRSSA) to assess video recordings of diagnostic shoulder arthroscopy performed on live patients 18 . The GRSSA demonstrated construct validity but poor inter-rater reliability 18 .…”

“…In a similar process, Hoyle et al created the Global Rating Scale for Shoulder Arthroscopy (GRSSA) to assess video recordings of diagnostic shoulder arthroscopy performed on live patients 18 . The GRSSA demonstrated construct validity but poor inter-rater reliability 18 . Due to their design, both of these global rating scales lack generalizability in that they can only be used to assess partial menisectomy and shoulder arthroscopy respectively.…”

Using the Arthroscopic Surgery Skill Evaluation Tool as a Pass-Fail Examination

“…There was considerable agreement between raters with an interclass correlation coefficient of 0.83 (Table III). Assessment tools similar to the ASSET are reliable for assessing arthroscopic skill 3,5 . However, neither assessment was developed or was tested as a pass-fail assessment 3,5 .…”

“…Assessment tools similar to the ASSET are reliable for assessing arthroscopic skill 3,5 . However, neither assessment was developed or was tested as a pass-fail assessment 3,5 . When initially developing the ASSET, we intended that it would be used as a pass-fail assessment because this was a major flaw in previously described arthroscopic assessments.…”

“…However, an assessment that demonstrates validity and reliability cannot be assumed to be adequate for use as a pass-fail examination. Published assessments of arthroscopic technical skill can demonstrate validity and reliability, but all fall short of being suitable for use as a pass-fail examination because of a lack of specific criteria for passing the assessment [2][3][4][5] . This shortcoming leaves any potential user with the task of subjectively defining what a passing score will be and limits the use of these tools.…”