Is There a Role for Fidelity Self-Assessment in the Individual Placement and Support Model of Supported Employment?

Abstract

Objective Fidelity assessments help ensure evidence-based practices are implemented properly. While typically conducted by independent raters, some programs have implemented self- assessments due to resource constraints. Self-assessments were compared to independent assessments within programs implementing Individual Placement and Support (IPS) supported employment. Methods Eleven community-based outpatient programs in New York State completed both self and independent assessments. ICCs and paired t-tests were used to compare independent and self-rated assessments. Results Mean scores are within the range of “fair fidelity” to IPS. Mean self (91.7) and independent (92.9) scores were not significantly different from one another. However, significant differences were found among individual items in this small sample. Conclusions and Implications for Practice Self-assessments may be valid for examining a program’s overall functioning, and useful when resource constraints prevent independent assessment. Independent assessors may be able to identify nuances, particularly among individual assessment items, that can identify areas for program improvement.

Using Instructional Design, Analyze, Design, Develop, Implement, and Evaluate, to Develop e-Learning Modules to Disseminate Supported Employment for Community Behavioral Health Treatment Programs in New York State

“…Notably, only half of the providers who completed the evaluation noted an intention to change their practice, and we did not have the capacity to assess practice change at the individual provider level at this stage of IPS implementation (level 3). However, in our subsequent work ( 27 ), program fidelity assessments using established measures demonstrated improvement over time, suggesting that level 3 provider practice change and fidelity self-assessed by program sites are shown to be associated with higher employment rates (level 4), which are sustained over time ( 28 ). Future research may focus on more rigorous evaluation of knowledge, practice change, mixed-method assessment of how the content from e-learning modules influences practice, and the essential role of care recipients in helping to design training within implementation efforts.…”

Relationship Between Self-Assessed Fidelity and Self-Reported Employment in the Individual Placement and Support Model of Supported Employment

“…Fidelity self-assessments are completed annually by participating programs. In a recent study comparing self-assessed fidelity to independent ratings (14), we found agreement between these methods when looking at programs’ total Scale score, though there was variability in agreement for individual items.…”

“…The fidelity assessments in our study are self-reported by program sites; this can be considered both an asset and a limitation. While independent, external assessment is an important accountability measure, it can be expensive and burdensome for sponsoring agencies to provide on an ongoing basis (14). Our prior study showed that agency self-report can yield similar fidelity scores to independent assessors within the NYS learning collaborative (14) and this study’s results indicate that self-reported fidelity, like independently-assessed fidelity (10, 11, 12), is associated with greater employment rates.…”

“…In the learning collaborative, program staff were taught about IPS fidelity generally, and how to conduct fidelity self-assessment specifically, through webinars and program-specific consultation calls and visits. They received 1 to 3 hours of specific fidelity training that was augmented by a focus on fidelity that was intertwined throughout the entire training experience each year (14). Fidelity self-assessments are completed annually by participating programs.…”

Mainstreaming Effective Employment Support for Individuals with Health Conditions: An Analytical Framework for the Effective Design of Modified Individual Placement and Support (IPS) Models

“…For traditional IPS services the fidelity scale is a powerful guide to designing and delivering high quality IPS services and performance. Given the activity, trialling and effectiveness of IPS services over the past twenty years, and given the central place of IPS fidelity to the IPS model, the IPS fidelity scale has understandably been the subject of significant attention within the academic literature in a range of ways including: the creation and evolution of the IPS-25 fidelity scale (Bond et al, 1997;Bond et al, 2002;McGrew and Griss, 2005;Bond et al, 2012b); the link between employment outcomes with the total fidelity score (Bond et al, 2012b;Kim et al, 2015;Lockett et al, 2016) and, as important but less commonly, individual fidelity items (Bond et al, 2012b;Kim et al, 2015;Margolies et al, 2018); sub-dimensions of the fidelity scale (Bond et al, 2002); differing levels or types of fidelity scale (Mowbray et al, 2003;Bond et al, 2011); fidelity assessment and self-assessment (Bond et al, 2011;Margolies et al, 2017); and the moderating role of individual and contextual factors between IPS fidelity and employment outcomes (Campbell et al, 2010;Metcalfe et al, 2018).…”