The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing knee joint mobilization skills of physical therapy students both on-site and off-site

Abstract

Background: There is a need to validate the reliability and validity of Direct Observation of Procedural Skills (DOPS) of knee joint mobilization for providing immediate feedback on the procedure. Additionally, it is crucial to examine whether the tools developed for on-site use can be applied off-site to prepare for future practice. The purpose of this study is to describe the implementation and clinimetric properties of a DOPS for knee joint mobilization evaluation (DOPSknee) on-site and off-site and to determine the pre-clinical competencies of physical therapy (PT) students at school. Methods: After the DOPSknee had been developed, 42 pre-clinical students from a single PT program were videotaped performing knee joint mobilization. Their performance was assessed in the standard manner by two clinical instructors using the DOPSknee. Assessments were conducted on-site and then off-site using the recorded videos after a time gap ranging from 1.5 months to 4.5 months. Results: The DOPSknee demonstrated a good level of on-site inter-rater reliability (intraclass correlation coefficient [ICC] = 0.78). It had a poor level of off-site inter-rater reliability (ICC = 0.37) and a poor to good level of intra-rater reliability between on-site and off-site evaluations (evaluator 1: ICC = 0.42; evaluator 2: ICC = 0.88). However, the DOPSknee was shown to have good reliability average measures between the on-site and off-site evaluations (average on-site vs. average off-site: ICC = 0.84). Average total DOPSknee scores were significantly correlated with the average global rating scores on-site and off-site (r= 0.47 and 0.75) and in cross-over conditions (r = 0.54 and 0.71). The cut-off score for the DOPSkneewas determined to be 14 points, and the construct validity analysis of both on-site and off-site evaluations from the average DOPSknee total scores demonstrated that the average global rating scores of the students who passed were significantly higher (p< 0.05). Conclusions: The clinimetric properties of the DOPSknee protocol revealed generally good results for on-site validity and reliability in assessing student performance and a need to be interpreted and applied with care off-site. The average measures may help improve the results from the two evaluators.