From Utopia Through Dystopia: Charting a Course for Learning Analytics in Competency-Based Medical Education

Abstract

The transition to the assessment of entrustable professional activities as part of competency-based medical education (CBME) has substantially increased the number of assessments completed on each trainee. Many CBME programs are having difficulty synthesizing the increased amount of assessment data. Learning analytics are a way of addressing this by systematically drawing inferences from large datasets to support trainee learning, faculty development, and program evaluation. Early work in this field has tended to emphasize the significant potential of analytics in medical education. However, concerns have been raised regarding data security, data ownership, validity, and other issues that could transform these dreams into nightmares. In this paper, the authors explore these contrasting perspectives by alternately describing utopian and dystopian futures for learning analytics within CBME. Seeing learning analytics as an important way to maximize the value of CBME assessment data for organizational development, they argue that their implementation should continue within the guidance of an ethical framework.

The Impact of Electronic Data to Capture Qualitative Comments in a Competency-Based Assessment System

“…With the advent of competency-based medical education (CBME), there has been a marked use of digital systems to capture WBA [ 19 - 22 ]. With the increasing use of these databases, many groups have resorted to trainee behaviors around data capture [ 23 , 24 ].…”

Developing a dashboard for program evaluation in competency-based training programs: a design-based research project

“…The senior author (BT) reviewed the literature on program evaluation in competency-based medical education, [13][14][15]17,18,35,36 learning analytics, 21,[37][38][39] and data visualization 25,31,37,40 to generate ideas for the initial iteration of the program evaluation dashboard. The initial interviews occurred with the program director, associate program director, institutional CBD Lead, postgraduate Dean.…”

“…Further investigation is also needed to identify analytics that warrant the flagging of program leaders. 39 For example, program directors might be interested in being informed if an EPA becomes dramatically under-represented or a rotation is significantly deviating from historical measures. Identifying appropriate alarms or flags and determining how this information should be relayed to program leadership would allow such features to be incorporated into the dashboard.…”

Competency-Based Medical Training in Anesthesiology: Has It Delivered on the Promise of Better Education?

“…44 Machine learning is also being explored to provide learning analytics to aggregated CBME assessment data to analyze and predict a trainee's EPA progress to support residents and programs. 45 As our experience deepens with CBME, more work is needed to demonstrate that the shift to CBME improves trainee performance and patient outcomes. Since operating room metrics are often used to assess the quality of patient care, one anesthesiology program in the United States examined perioperative databases to assess the efficiency of resident performance.…”