Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study

Abstract

Introduction Podcasts are increasingly being used for medical education. Studies have found that the assessment of the quality of online resources can be challenging. We sought to determine the reliability of gestalt quality assessment of education podcasts in emergency medicine. Methods An international, interprofessional sample of raters was recruited through social media, direct contact, and the extended personal network of the study team. Each participant listened to eight podcasts (selected to include a variety of accents, number of speakers, and topics) and rated the quality of that podcast on a seven-point Likert scale. Phi coefficients were calculated within each group and overall. Decision studies were conducted using a phi of 0.8. Results A total of 240 collaborators completed all eight surveys and were included in the analysis. Attendings, medical students, and physician assistants had the lowest individual-level variance and thus the lowest number of required raters to reliably evaluate quality (phi >0.80). Overall, 20 raters were required to reliably evaluate the quality of emergency medicine podcasts. Discussion Gestalt ratings of quality from approximately 20 health professionals are required to reliably assess the quality of a podcast. This finding should inform future work focused on developing and validating tools to support the evaluation of quality in these resources.

Evaluating Effectiveness of YouTube Videos for Teaching Medical Students CPR: Solution to Optimizing Clinician Educator Workload during the COVID-19 Pandemic

“…Additionally, YouTube is reported to be the most commonly used resource for medical trainees to learn basic procedural skills and satisfies the learning preference of millennials [ 21 , 22 ]. Although it is still arguable whether the quality, accuracy, and updated content of the FOAMed resources are appropriately monitored [ 11 , 12 ], measurement tools evaluating the quality of FOAMed resources have been developed and have the potential to address their rigor, reliability, and efficacy [ 13 , 14 , 15 ]. However, it has still not been clarified whether using existing YouTube videos on CPR can serve as an alternative to creating lecture videos for preclinical medical students.…”

“…Videos explaining and demonstrating standardized resuscitation protocols have long been used in cardiopulmonary resuscitation (CPR) courses. While previous studies reported that the quality of some of the FOAMed resources is inadequate [ 11 , 12 ], new criteria ensuring the educational quality of such materials have been established and used to optimize the educational opportunities for trainees even during the COVID-19 pandemic [ 13 , 14 , 15 ]. Therefore, the application of quality-assured FOAMed CPR videos for preclinical resuscitation training of medical students has the potential to be an alternative instructional methodology during the COVID-19 pandemic.…”

SAEM systematic online academic resource (SOAR) review: Gastrointestinal illnesses

“…Evaluation by gestalt alone, particularly by smaller numbers of raters, has been shown to be unreliable. [10][11][12] The Society for Academic Emergency Medicine (SAEM's) systematic online academic resource (SOAR) series has aimed to comprehensively evaluate and curate FOAM resources within a specific EM category utilizing the revised Medical Education Translational Resources: Impact and Q uality (rMETRIQ ) score. [13][14][15] The rMETRIQ score (Table 1) is based on three main domains: content, credibility, and peer review, which are assessed using seven quality-related questions on a 4-point scale for a maximal 21 points.…”

“…Given these criticisms of FOAM, it is imperative for educators to guide trainees to more reliable and reputable resources. Evaluation by gestalt alone, particularly by smaller numbers of raters, has been shown to be unreliable 10–12 …”