Three out of four published systematic reviews on COVID-19 treatments were not registered and one-third of those registered were published: a meta-research study

Siemens, Waldemar; Nothacker, Julia; Stadelmaier, Julia; Meerpohl, Jörg J.; Schmucker, Christine

doi:10.1016/j.jclinepi.2022.09.011

Cited by 10 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various protagonists of the EbM community may argue that adapting EbM to fast-changing environments via methods accelerating knowledge processing is dangerous in sacrificing quality for speed. [124][125][126][127] We agree in part with this argument. However, the advantage of EbM is its goal to deliver high-quality results using rigorous methods.…”

Section: Discussionsupporting

confidence: 60%

See 1 more Smart Citation

Reducing uncertainty in evidence‐based health policy by integrating empirical and theoretical evidence: An EbM+theory approach

Pfaff

Schmitt

2023

Evaluation Clinical Practice

View full text Add to dashboard Cite

BackgroundTo reduce their decisional uncertainty, health policy decision‐makers rely more often on experts or their intuition than on evidence‐based knowledge, especially in times of urgency. However, this practice is unacceptable from an evidence‐based medicine (EbM) perspective. Therefore, in fast‐changing and complex situations, we need an approach that delivers recommendations that serve decision‐makers' needs for urgent, sound and uncertainty‐reducing decisions based on the principles of EbM.AimsThe aim of this paper is to propose an approach that serves this need by enriching EbM with theory.Materials and MethodsWe call this the EbM+theory approach, which integrates empirical and theoretical evidence in a context‐sensitive way to reduce intervention and implementation uncertainty.ResultsWithin this framework, we propose two distinct roadmaps to decrease intervention and implementation uncertainty: one for simple and the other for complex interventions. As part of the roadmap, we present a three‐step approach: applying theory (step 1), conducting mechanistic studies (EbM+; step 2) and conducting experiments (EbM; step 3).DiscussionThis paper is a plea for integrating empirical and theoretical knowledge by combining EbM, EbM+ and theoretical knowledge in a common procedural framework that allows flexibility even in dynamic times. A further aim is to stimulate a discussion on using theories in health sciences, health policy, and implementation.ConclusionThe main implications are that scientists and health politicians – the two main target groups of this paper—should receive more training in theoretical thinking; moreover, regulatory agencies like NICE may think about the usefulness of integrating elements of the EbM+theory approach into their considerations.

show abstract

Section: Discussionsupporting

confidence: 60%

“…Various protagonists of the EbM community may argue that adapting EbM to fast‐changing environments via methods accelerating knowledge processing is dangerous in sacrificing quality for speed 124–127 . We agree in part with this argument.…”

Section: Discussionmentioning

confidence: 59%

Reducing uncertainty in evidence‐based health policy by integrating empirical and theoretical evidence: An EbM+theory approach

Pfaff

Schmitt

2023

Evaluation Clinical Practice

View full text Add to dashboard Cite

show abstract

“…We expect that especially additional IfR elements (e.g., biomarker) could vary depending on the medical field. A similar IfR analysis for a sample of systematic reviews assessing treatments for COVID-19 [ 21 ] is underway and will add important information regarding consistency and generalizability of the findings in the present study.…”

Section: Discussionmentioning

confidence: 99%

Evaluation of ‘implications for research’ statements in systematic reviews of interventions in advanced cancer patients – a meta-research study

Siemens,

Bantle,

Ebner

et al. 2023

BMC Med Res Methodol

View full text Add to dashboard Cite

Background Implications for research (IfR) sections are an important part of systematic reviews (SRs) to inform health care researchers and policy makers. PRISMA 2020 recommends reporting IfR, while Cochrane Reviews require a separate chapter on IfR. However, it is unclear to what extent SRs discuss IfR. We aimed i) to assess whether SRs include an IfR statement and ii) to evaluate which elements informed IfR statements. Methods We conducted a meta-research study based on SRs of interventions in advanced cancer patients from a previous project (CRD42019134904). As suggested in the Cochrane Handbook, we assessed if the following predefined variables were referred to in IfR statements: patient, intervention, control, outcome (PICO) and study design; concepts underlying Grading of Recommendations, Assessment, Development and Evaluation (GRADE) domains: risk of bias, inconsistency, indirectness, imprecision, publication bias. Data were independently extracted by three reviewers after piloting the data extraction form. Discrepancies were resolved in weekly in-depth discussions. Results We included 261 SRs. The majority evaluated a pharmacological intervention (n = 244, 93.5%); twenty-nine were Cochrane Reviews (11.1%). Four out of five SRs included an IfR statement (n = 210, 80.5%). IfR statements commonly addressed ‘intervention’ (n = 121, 57.6%), ‘patient ‘ (n = 113, 53.8%), and ‘study design’ (n = 107, 51.0%). The most frequent PICO and study design combinations were ‘patient and intervention ‘ (n = 71, 33.8%) and ‘patient, intervention and study design ‘ (n = 34, 16.2%). Concepts underlying GRADE domains were rarely used for informing IfR recommendations: ‘risk of bias ‘ (n = 2, 1.0%), and ‘imprecision ‘ (n = 1, 0.5%), ‘inconsistency ‘ (n = 1, 0.5%). Additional elements informing IfR were considerations on cost effectiveness (n = 9, 4.3%), reporting standards (n = 4, 1.9%), and individual patient data meta-analysis (n = 4, 1.9%). Conclusion Although about 80% of SRs included an IfR statement, the reporting of PICO elements varied across SRs. Concepts underlying GRADE domains were rarely used to derive IfR. Further work needs to assess the generalizability beyond SRs in advanced cancer patients. We suggest that more specific guidance on which and how IfR elements to report in SRs of interventions needs to be developed. Utilizing PICO elements and concepts underlying GRADE according to the Cochrane Handbook to state IfR seems to be a reasonable approach in the interim. Registration CRD42019134904.

show abstract

“…We expect that especially additional IfR elements (e.g., biomarker) could vary depending on the medical eld. A similar IfR analysis for a sample of systematic reviews assessing treatments for COVID-19 [21] is underway and will add important information regarding consistency and generalizability of the ndings in the present study.…”

Section: Limitationsmentioning

confidence: 93%

Evaluation of ‘implications for research’ sections in systematic reviews of interventions in advanced cancer patients – a meta-research study

Siemens

Bantle

Ebner

et al. 2023

Preprint

View full text Add to dashboard Cite

Background Implications for research (IfR) sections are an important part of systematic reviews (SRs) in healthcare. The PRISMA 2020 statement recommends discussion of the implications of the results for practice, while Cochrane Reviews require an IfR section. However, it is unclear to what extent systematic reviews discuss IfR and if this section is useful. We aimed i) to assess whether systematic reviews include an IfR section and ii) to evaluate which elements informed reporting of IfR. Methods We conducted a meta-research study based on SRs of interventions in advanced cancer patients from a previous project (CRD42019134904). We assessed if the following predefined IfR variables were referred to in the included SRs: patients, intervention, control, outcome, study design (PICOS); Grading of Recommendations, Assessment, Development and Evaluation (GRADE) domains: risk of bias, inconsistency, indirectness, imprecision, publication bias. Data were independently extracted by three reviewers after piloting the data extraction form. Discrepancies were resolved in weekly in-depth discussions. Results We included 261 SRs. The majority evaluated a pharmacological intervention (244, 93.5%); twenty-nine were Cochrane Reviews (11.1%). Four out of five SRs included an IfR section (210, 80.5%). PICOS elements commonly addressed in IfR sections were ‘intervention’ (121, 57.6%), ‘patient‘ (113, 53.8%) and ‘study design’ (107, 51.0%). The most frequent PICOS combinations were ‘patient and intervention‘ (71, 33.8%) and ‘patient, intervention and study design‘ (34, 16.2%). GRADE domains were rarely used for informing IfR recommendations: ‘risk of bias‘ (2, 1.0%), and ‘imprecision‘ (1, 0.5%), ‘inconsistency‘ (1, 0.5%). Additional factors informing IfR recommendations were considerations on cost effectiveness (9, 4.3%), reporting standards (4, 1.9%), and individual patient data meta-analysis (4, 1.9%). Conclusion Although four out of five systematic reviews of our sample included an IfR section, these were mostly considering only selected PICOS elements and GRADE domains were used rarely to guide future research recommendations. In the absence of established reporting standards, we propose more comprehensive and structured IfR sections in SRs informed by PICOS elements and GRADE domains. We suggest further empirical and theoretical work to eventually develop guidance on what IfR elements to consider and how to report IfR in SRs of interventions.

show abstract

Three out of four published systematic reviews on COVID-19 treatments were not registered and one-third of those registered were published: a meta-research study

Cited by 10 publications

References 30 publications

Reducing uncertainty in evidence‐based health policy by integrating empirical and theoretical evidence: An EbM+theory approach

Reducing uncertainty in evidence‐based health policy by integrating empirical and theoretical evidence: An EbM+theory approach

Evaluation of ‘implications for research’ statements in systematic reviews of interventions in advanced cancer patients – a meta-research study

Evaluation of ‘implications for research’ sections in systematic reviews of interventions in advanced cancer patients – a meta-research study

Contact Info

Product

Resources

About