Twelve tips to promote successful development of a learner performance dashboard within a medical education program

Boscardin, Christy; Fergus, Kirkpatrick B.; Hellevig, Bonnie; Hauer, Karen E.

doi:10.1080/0142159x.2017.1396306

Cited by 46 publications

(57 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The current lack of systematic procedures for evidentiary review, weighting and interpretation and the heterogeneity of practices within committees have been problematised . In response, a variety of strategies have been proposed to standardise CCCs, including best operating practices, faculty training, data visualisation software and data analysis programs …”

Section: Introductionmentioning

confidence: 99%

Some assembly required: tracing the interpretative work of Clinical Competency Committees

et al. 2019

View full text Add to dashboard Cite

Objectives This qualitative study describes the social processes of evidence interpretation employed by Clinical Competency Committees (CCCs), explicating how they interpret, grapple with and weigh assessment data. Methods Over 8 months, two researchers observed 10 CCC meetings across four postgraduate programmes at a Canadian medical school, spanning over 25 hours and 100 individual decisions. After each CCC meeting, a semi‐structured interview was conducted with one member. Following constructivist grounded theory methodology, data collection and inductive analysis were conducted iteratively. Results Members of the CCCs held an assumption that they would be presented with high‐quality assessment data that would enable them to make systematic and transparent decisions. This assumption was frequently challenged by the discovery of what we have termed ‘problematic evidence’ (evidence that CCC members struggled to meaningful interpret) within the catalogue of learner data. When CCCs were confronted with ‘problematic evidence’, they engaged in lengthy, effortful discussions aided by contextual data in order to make meaning of the evidence in question. This process of effortful discussion enabled CCCs to arrive at progression decisions that were informed by, rather than ignored, problematic evidence. Conclusions Small groups involved in the review of trainee assessment data should be prepared to encounter evidence that is uncertain, absent, incomplete, or otherwise difficult to interpret, and should openly discuss strategies for addressing these challenges. The answer to the problem of effortful processes of data interpretation and problematic evidence is not as simple as generating more data with strong psychometric properties. Rather, it involves grappling with the discrepancies between our interpretive frameworks and the inescapably subjective nature of assessment data and judgement.

show abstract

Section: Introductionmentioning

confidence: 99%

Some assembly required: tracing the interpretative work of Clinical Competency Committees

et al. 2019

View full text Add to dashboard Cite

show abstract

“…We utilized a design-based research approach [24][25][26] and followed best practices 23,26,27 to meet this objective. We report the qualitative components of our project in keeping with the consolidated criteria for reporting qualitative research.…”

Section: Methodsmentioning

confidence: 99%

“…17,22 The development of such dashboards is an iterative process 18 and requires collaboration with information technology experts, assessment experts, data managers, and dashboard users. 23…”

Section: Introductionmentioning

confidence: 99%

Developing a dashboard to meet Competence Committee needs: a design-based research project

et al. 2020

View full text Add to dashboard Cite

Background: Competency-based programs are being adopted in medical education around the world. Competence Committees must visualize learner assessment data effectively to support their decision-making. Dashboards play an integral role in decision support systems in other fields. Design-based research allows the simultaneous development and study of educational environments. Methods: We utilized a design-based research process within the emergency medicine residency program at the University of Saskatchewan to identify the data, analytics, and visualizations needed by its Competence Committee, and developed a dashboard incorporating these elements. Narrative data were collected from two focus groups, five interviews, and the observation of two Competence Committee meetings. Data were qualitatively analyzed to develop a thematic framework outlining the needs of the Competence Committee and to inform the development of the dashboard. Results: The qualitative analysis identified four Competence Committee needs (Explore Workplace-Based Assessment Data, Explore Other Assessment Data, Understand the Data in Context, and Ensure the Security of the Data). These needs were described with narratives and represented through visualizations of the dashboard elements. Conclusions: This work addresses the practical challenges of supporting data-driven decision making by Competence Committees and will inform the development of dashboards for programs, institutions, and learner management systems.

show abstract

“…7 Data visualisation involves the creation and study of data that have been abstracted, summarised or categorised in some schematic form. 7,8 It combines data from various assessments to organise and visualise data for interpretation, and is employed to effectively provide feedback to students and to assist faculty members in making competency-based decisions. 4,6 The data visualisation can provide efficient feedback, yet Hauer and colleagues found that students' use of this information to guide learning is variable: some students engaged in feedback to improve performance, whereas others did not.…”

Section: Assessment Reportingmentioning

confidence: 99%

“…Boscardin and colleagues' 12 tips to promote successful development of a learner performance dashboard provides excellent guidance. 7 There were lessons learned during ... it is important to pay attention to the alignment with the construct of competency and to engage stakeholders ...…”

Section: Recommendations For Cbme Assessment Data Visualisationmentioning

confidence: 99%

Creating a learner performance dashboard for programmatic assessment

Santen

Myklebust

Cabrera

et al. 2019

The Clinical Teacher

View full text Add to dashboard Cite

Editor's note: Programmatic assessment is a process that is gaining traction in medical and health professional education. It is an approach that aligns with competency-based education. Routine and longitudinal data are collected about learners' competence, and their progress is regularly reviewed and analysed. Typically, a variety of assessment methods are used with the overall aim of allowing both learners and educators to gain an understanding of the extent of learning and to maximise feedback for optimal educational impact (assessment for learning). Eventually the process may be used for high-stakes decisions at the end of a programme or phase of study (assessment of learning). In this toolbox article the authors describe how they have developed a method of visually representing medical student attainment during programmatic assessment on a dashboard. This development required input from educators, faculty members and information technology staff. The graphs obtained can help students and staff to see how a student is performing over time. The authors share feedback on the approach from learners and faculty members and provide recommendations for readers who may wish to develop something similar.

show abstract

Twelve tips to promote successful development of a learner performance dashboard within a medical education program

Cited by 46 publications

References 18 publications

Some assembly required: tracing the interpretative work of Clinical Competency Committees

Some assembly required: tracing the interpretative work of Clinical Competency Committees

Developing a dashboard to meet Competence Committee needs: a design-based research project

Creating a learner performance dashboard for programmatic assessment

Contact Info

Product

Resources

About