Enhancing the impact of BEME systematic reviews on educational practice

Gordon, Morris; Carneiro, António Vaz; Patrício, Madalena

doi:10.3109/0142159x.2015.1042437

Cited by 5 publications

(5 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…'The difficulty of evaluating any educational philosophy in a scientific manner' occupied the early days of seeking best evidence (available) in medical (and other health professions) education [14, p.1]. The push to widen the horizons of BEME reviews has continued [15]. Systematic reviews in health services research and public health have long since been dealing with condensing complexity into concise counsel.…”

Section: Building Beyond Kirkpatrick and 'What Work?' Using An Examplementioning

confidence: 99%

Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Maudsley

Taylor

2020

Medical Education Online

View full text Add to dashboard Cite

Background: Systematic reviews in health professions education may well under-report struggles to synthesize disparate evidence that defies standard quantitative approaches. This paper reports further process analysis in a previously reported systematic review about mobile devices on clinical placements. Objective: For a troublesome systematic review: (1) Analyse further the distribution and reliability of classifying the evidence to Maxwell quality dimensions (beyond 'Does it work?') and their overlap with Kirkpatrick K-levels. (2) Analyse how the abstracts represented those dimensions of the evidence-base. (3) Reflect on difficulties in synthesis and merits of Maxwell dimensions. Design: Following integrative synthesis of 45 K2-K4 primary studies (by combined contentthematic analysis in the pragmatism paradigm): (1) Hierarchical cluster analysis explored overlap between Maxwell dimensions and K-levels. Independent and consensus-coding to Maxwell dimensions compared (using: percentages; kappa; McNemar hypothesis-testing) prevs post-discussion and (2) article abstract vs main body. (3) Narrative summary captured process difficulties and merits. Results: (1) The largest cluster (five-cluster dendrogram) was acceptability-accessibility-K1appropriateness-K3, with K1 and K4 widely separated. For article main bodies, independent coding agreed most for appropriateness (good; adjusted kappa = 0.78). Evidence increased significantly pre-post-discussion about acceptability (p = 0.008; 31/45→39/45), accessibility, and equity-ethics-professionalism. (2) Abstracts suggested efficiency significantly less than main bodies evidenced: 31.1% vs 44.4%, p = 0.031. 3) Challenges and merits emerged for before, during, and after the review. Conclusions: There should be more systematic reporting of process analysis about difficulties synthesizing suboptimal evidence-bases. In this example, Maxwell dimensions were a useful framework beyond K-levels for classifying and synthesizing the evidence-base.

show abstract

Section: Building Beyond Kirkpatrick and 'What Work?' Using An Examplementioning

confidence: 99%

Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Maudsley

Taylor

2020

Medical Education Online

View full text Add to dashboard Cite

show abstract

“…In health professions education (HPE), researchers have argued that knowledge syntheses are as important as primary studies [ 1 ]. Thus, it is unsurprising that their numbers have been on the rise, with most HPE journals accepting knowledge syntheses, and foundations and professional groups offering funding for their creation (e.g., the Gold Foundation).…”

Section: Introductionmentioning

confidence: 99%

“…Despite support for their creation and swelling numbers [ 2 ], knowledge syntheses, especially systematic reviews, have been criticized not only by those who try to use them, but also by those who publish them [ 1 , 3 – 5 ]. Norman argued that many reviews end up as exercises in ‘bean counting’ and often fail to provide conclusive, usable evidence for practitioners [ 6 ].…”

Section: Introductionmentioning

confidence: 99%

“…Norman argued that many reviews end up as exercises in ‘bean counting’ and often fail to provide conclusive, usable evidence for practitioners [ 6 ]. Furthermore, Gordon suggested that systematic reviews often lack relevance and fail to incorporate educational elements critical for educational practice and policy [ 1 ]. Despite these criticisms, our field has yet to evaluate HPE knowledge syntheses to determine whether or not they are ready for translation into educational practice and policy.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Examining the readiness of best evidence in medical education guides for integration into educational practice: A meta-synthesis

Maggio¹,

Thomas²,

Chen³

et al. 2018

Perspect Med Educ

View full text Add to dashboard Cite

BackgroundTo support evidence-informed education, health professions education (HPE) stakeholders encourage the creation and use of knowledge syntheses or reviews. However, it is unclear if these knowledge syntheses are ready for translation into educational practice. Without understanding the readiness, defined by three criteria—quality, accessibility and relevance—we risk translating weak evidence into practice and/or providing information that is not useful to educators.MethodsA librarian searched Web of Science for knowledge syntheses, specifically Best Evidence in Medical Education (BEME) Guides. This meta-synthesis focuses on BEME Guides because of their explicit goal to inform educational practice and policy. Two authors extracted data from all Guides, guided by the 25-item STructured apprOach to the Reporting In healthcare education of Evidence Synthesis (STORIES).ResultsForty-two Guides published in Medical Teacher between 1999 and 2017 were analyzed. No Guide met all STORIES criteria, but all included structured summaries and most described their literature search (n = 39) and study inclusion/exclusion (n = 40) procedures. Eleven Guides reported the presence of theory and/or educational principles, and eight consulted with external subject matter experts. Accessibility to each Guide’s full-text and supplemental materials was variable.DiscussionFor a subset of HPE knowledge syntheses, BEME Guides, this meta-synthesis identifies factors that support readiness and indicates potential areas of improvement, such as consistent access to Guides and inclusion of external subject matter experts on the review team. This analysis is useful for understanding the current readiness of HPE knowledge syntheses and informing future reviews to evolve so they can catalyze translation of evidence into educational practice.

show abstract

“…Despite support for their creation and swelling numbers [2], knowledge syntheses, especially systematic reviews, have been criticized not only by those who try to use them, but also by those who publish them [1,[3][4][5]. Norman argued that many reviews end up as exercises in 'bean counting' and often fail to provide conclusive, usable evidence for practitioners [6].…”

Section: Introductionmentioning

confidence: 99%

Examining the readiness of Best Evidence in Medical Education Guides for integration into educational practice: A meta-synthesis

Maggio¹,

Thomas²,

Chen³

et al. 2018

Preprint

View full text Add to dashboard Cite

To support evidence-informed education, health professions education (HPE) stakeholders encourage the creation and use of knowledge syntheses or reviews. However, it is unclear if these knowledge syntheses are ready for translation into educational practice. Without understanding the readiness, defined by three criteria, quality, accessibility and relevance, we risk translating weak evidence into practice and/or providing information to educators that is not usable, both of which have the potential to frustrate teachers and learners and impair teaching and learning practices.Methods: A librarian searched for knowledge syntheses, specifically Best Evidence in Medical Education (BEME) Guides, using Web of Science. Two authors extracted data from all Guides, guided by the 25-item STructured apprOach to the Reporting In healthcare education of Evidence Synthesis (STORIES). Results: Forty-two Guides published in Medical Teacher between 1999 and 2017 were analyzed. No Guide met all STORIES criteria, but all included structured summaries and most described their literature search (n=39) and study inclusion/exclusion (n=40) procedures. Eleven Guides reported the presence of theory and/or educational principles, and eight consulted with external subject matter experts. Accessibility to each Guide’s full-text and supplemental materials was variable. Discussion: For a subset of HPE knowledge syntheses, BEME Guides, this meta-synthesis identifies factors that support readiness and indicates potential areas of improvement, such as consistent access to Guides and inclusion of external subject matter experts. This analysis is useful for understanding the current readiness of HPE knowledge syntheses and informing future reviews to evolve so they can catalyze translation of evidence into educational practice.

show abstract

Enhancing the impact of BEME systematic reviews on educational practice

Cited by 5 publications

References 6 publications

Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Examining the readiness of best evidence in medical education guides for integration into educational practice: A meta-synthesis

Examining the readiness of Best Evidence in Medical Education Guides for integration into educational practice: A meta-synthesis

Contact Info

Product

Resources

About

Enhancing the impact of BEME systematic reviews on educational practice

Cited by 5 publications

References 6 publications

Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Examining the readiness of best evidence in medical education guides for integration into educational practice: A&nbsp;meta-synthesis

Examining the readiness of Best Evidence in Medical Education Guides for integration into educational practice: A meta-synthesis

Contact Info

Product

Resources

About

Examining the readiness of best evidence in medical education guides for integration into educational practice: A meta-synthesis