Competency‐based education calls for programmatic assessment: But what does this look like in practice?

Abstract

Rationale, aims, and objectivesProgrammatic assessment has been identified as a system‐oriented approach to achieving the multiple purposes for assessment within Competency‐Based Medical Education (CBME, i.e., formative, summative, and program improvement). While there are well‐established principles for designing and evaluating programs of assessment, few studies illustrate and critically interpret, what a system of programmatic assessment looks like in practice. This study aims to use systems thinking and the ‘two communities’ metaphor to interpret a model of programmatic assessment and to identify challenges and opportunities with operationalization.MethodAn interpretive case study was used to investigate how programmatic assessment is being operationalized within one competency‐based residency program at a Canadian university. Qualitative data were collected from residents, faculty, and program leadership via semi‐structured group and individual interviews conducted at nine months post‐CBME implementation. Data were analyzed using a combination of data‐based inductive analysis and theory‐derived deductive analysis.ResultsIn this model, Academic Advisors had a central role in brokering assessment data between communities responsible for producing and using residents' performance information for decision making (i.e., formative, summative/evaluative, and program improvement). As system intermediaries, Academic Advisors were in a privileged position to see how the parts of the assessment system contributed to the functioning of the whole and could identify which system components were not functioning as intended. Challenges were identified with the documentation of residents' performance information (i.e., system inputs); use of low‐stakes formative assessments to inform high‐stakes evaluative judgments about the achievement of competence standards; and gaps in feedback mechanisms for closing learning loops.ConclusionsThe findings of this research suggest that program stakeholders can benefit from a systems perspective regarding how their assessment practices contribute to the efficacy of the system as a whole. Academic Advisors are well positioned to support educational development efforts focused on overcoming challenges with operationalizing programmatic assessment.

Where the rubber meets the road — An integrative review of programmatic assessment in health care professions education

“…To ensure quality, it seemed counterproductive to enforce top-down decisions and mandatory requirements on how much or what type of assessment information should be used for learning and decisionmaking. This approach likely results in tick-box activities and in both learners and teachers gaming or Table 2 Inferred strategies from the literature to improve the value and use of programmatic assessment Inferred strategy and exemplifying references Build on creating a shared understanding of programmatic assessment by clearly introducing the nature and purpose, providing explanatory guidelines for individual assessments and how they are used in the system as a whole, and involving teachers and learners in the whole chain of the system [16,19,21,29,30,32,38,40] Provide teachers and learners with feedback on the quality of provided assessment information and how their input contributes to the decisionmaking process [17,21,24,40] Normalize daily feedback, observation, and follow-up, as well as reflection and continuous improvement [19,21,22,28,34,38] Be cautious with mandatory requirements, being overly bureaucratic, and the use of summative signals in the design of programmatic assessment [17, 20-22, 24, 28, 33-35, 40], but keep the approach flexible, fit for purpose and negotiable, specifically in relation to the information needs of different stakeholders and the realities of the educational context [16,17,20,21,24,28,33,34,41] Promote learner agency and the development of life-long learner capabilities by increasing learners' ownership over the assessment process [20,28,30,34,41] Address learners' and teachers' assessment beliefs and the implications o...…”

“…Although the introduction of promotion or clinical competency committees was found to enable or improve the high-stakes decisionmaking [19,30,40], learners thought their mentors would be more credible in making this decision than such committees [16] and supervisors often opted to rely on their own observations to inform their highstakes decisions [35]. Making a high-stakes decision depended on the quality of the available information [19,35,40], and this quality was often found to be poor or even problematic [17, 20-22, 27, 34, 35, 40].…”

Clinical teaching of university-degree nursing students: Are the nurses in practice in Uganda ready?

“…In their argument for competency-based nursing education (CBE), Gruppen, Mangrulkar, and Kolars (29) explain that CBE is driven by professional competencies that enable the integration of clinical practice and clinically driven assessments within a nursing programme. Integrating CBE in the BScN programme at the Muni University will inevitably force nurse educators to be updated with the current clinical practice, engage in best available evidence and drive for clinically driven assessments (30). In addition, this would improve the presence of nurse educators within the clinical practice environment hence improving the quality of clinical education for undergraduate nursing students.…”

Clinical teaching of undergraduate nursing students: Are the nurses in practice in Uganda ready?

“…In their argument for competency-based nursing education (CBE), Gruppen, Mangrulkar, and Kolars (30) explain that CBE is driven by professional competencies that enable the integration of clinical practice and clinically driven assessments within a nursing programme. Integrating CBE in the BScN programme at the Muni University will inevitably force nurse educators to be updated with the current clinical practice, engage in best available evidence and drive for clinically driven assessments (31). In addition, this would improve the presence of nurse educators within the clinical practice environment hence improving the quality of clinical education for undergraduate nursing students.…”