Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Abstract

Background: Systematic reviews in health professions education may well under-report struggles to synthesize disparate evidence that defies standard quantitative approaches. This paper reports further process analysis in a previously reported systematic review about mobile devices on clinical placements. Objective: For a troublesome systematic review: (1) Analyse further the distribution and reliability of classifying the evidence to Maxwell quality dimensions (beyond 'Does it work?') and their overlap with Kirkpatrick K-levels. (2) Analyse how the abstracts represented those dimensions of the evidence-base. (3) Reflect on difficulties in synthesis and merits of Maxwell dimensions. Design: Following integrative synthesis of 45 K2-K4 primary studies (by combined contentthematic analysis in the pragmatism paradigm): (1) Hierarchical cluster analysis explored overlap between Maxwell dimensions and K-levels. Independent and consensus-coding to Maxwell dimensions compared (using: percentages; kappa; McNemar hypothesis-testing) prevs post-discussion and (2) article abstract vs main body. (3) Narrative summary captured process difficulties and merits. Results: (1) The largest cluster (five-cluster dendrogram) was acceptability-accessibility-K1appropriateness-K3, with K1 and K4 widely separated. For article main bodies, independent coding agreed most for appropriateness (good; adjusted kappa = 0.78). Evidence increased significantly pre-post-discussion about acceptability (p = 0.008; 31/45→39/45), accessibility, and equity-ethics-professionalism. (2) Abstracts suggested efficiency significantly less than main bodies evidenced: 31.1% vs 44.4%, p = 0.031. 3) Challenges and merits emerged for before, during, and after the review. Conclusions: There should be more systematic reporting of process analysis about difficulties synthesizing suboptimal evidence-bases. In this example, Maxwell dimensions were a useful framework beyond K-levels for classifying and synthesizing the evidence-base.

The Kirkpatrick model for training evaluation: bibliometric analysis after 60 years (1959–2020)

“…Considering the extracted data set to show the most relevant domains in which Kirkpatrick's model is used, as seen in Figure 16, the most common domain for research using the Kirkpatrick model was medical training evaluation, both in all fields and in social sciences. Moreau (2017) and Maudsley and Taylor (2020) confirm that the Kirkpatrick model has been popular for straightforward, practical-toevaluate medical education programs.…”

Evaluation of the Effectiveness of Online Training to Improve Lecturers' Ability in E-Learning Management Using the Kirkpatrick Model

“…This study aims to analyze the effectiveness of online training programs to improve lecturers' ability to manage e-learning. The evaluation is carried out using the Kirkpatrick evaluation model which includes 4 (four) levels, namely 1) reaction; 2) learning; 3) behavior; 4) results (Maudsley & Taylor, 2020). This first level measures participants' reactions to the training and its related aspects (Jain, Sharma, & Shrivastava, 2021;Ragsdale et al, 2020).…”

Malay Local Wisdom Values in Elementary School Learning

“…Education is the process by which society, through educational institutions, intentionally transforms its cultural heritage, namely knowledge, values, and skills (Aragonés-González et al, 2020;Lyon, 2021;Maudsley & Taylor, 2020). Education aims to form a cultured human character and create humans who have broad insights and intelligence (Nuryanti, 2019;Van Nuland et al, 2020).…”