Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Nguyen, Phi‐Yen; Kanukula, Raju; McKenzie, Joanne E.; Alqaidoom, Zainab; Brennan, Sue; Haddaway, Neal R.; Hamilton, Daniel G; Karunananthan, Sathya; McDonald, Steve; Moher, David; Nakagawa, Shinichi; Nunan, David; Tugwell, Peter; Welch, Vivian; Page, Matthew J.

doi:10.1101/2022.04.11.22273688

Cited by 8 publications

(14 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other types of review materials that provide details of the data preparation and statistical analyses, including analytic code, files showing data conversion and imputations, and unprocessed extracted data, were not as frequently shared. This infrequent sharing of these review materials has been reported in other studies [10, 39, 43]. It is possible that some participants did not report sharing analytic code simply because their meta‐analytic software do not require coding (e.g., Review Manager or Comprehensive Meta‐Analysis).…”

Section: Discussionsupporting

confidence: 56%

“…Mandates from major public funders, such as one recently issued by the US National Institutes of Health (NIH) [5], are expected to accelerate the normalization of data sharing practices. However, as these initiatives tend to target clinical trials and basic science [6,7], the evidence synthesis research landscape is still plagued by low availability of data and code [8][9][10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Systematic reviewers' perspectives on sharing review data, analytic code, and other materials: A survey

Nguyen

McKenzie

Hamilton

et al. 2023

Cochrane Evidence Synthesis and Methods

Self Cite

View full text Add to dashboard Cite

Background: There are many benefits of sharing data, analytic code, and other materials, yet these items are infrequently shared among systematic reviews (SRs). It is unclear which factors influence authors' decisions to share data, code, or materials when publishing their SRs. Therefore, we aimed to explore systematic reviewers' perspectives on the importance of sharing review materials and factors that might influence such practices. Methods:We searched PubMed for SRs published from January to April 2021, from which we randomly allocated 50% to this survey and 50% to another survey on the replication of SRs. We sent an electronic survey to authors of these SRs (n = 4671) using Qualtrics. Quantitative responses were summarized using frequency analysis. Free-text answers were coded using an inductive approach.

show abstract

Section: Discussionsupporting

confidence: 56%

Section: Introductionmentioning

confidence: 99%

Systematic reviewers' perspectives on sharing review data, analytic code, and other materials: A survey

Nguyen

McKenzie

Hamilton

et al. 2023

Cochrane Evidence Synthesis and Methods

Self Cite

View full text Add to dashboard Cite

show abstract

“…We anticipate the same for PRISMA 2020 93 given its copublication in multiple high-impact journals. However, to date, there is a lack of strong evidence for an association between improved systematic review reporting and endorsement of PRISMA 2009 standards 43,111 . Most journals require a PRISMA checklist accompany submissions of systematic review manuscripts.…”

Section: Uptake and Impactmentioning

confidence: 99%

“…These have concentrated on evidence searching 40 and selection given how demanding it is for humans to maintain truly up-to-date evidence 2,41 . Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) 2 and studies related to COVID-19 42 , but such tools are not yet commonly used 43 . The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.…”

Section: Influences On the State Of Evidence Synthesismentioning

confidence: 99%

Guidance to Best Tools and Practices for Systematic Reviews

2023

View full text Add to dashboard Cite

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

show abstract

“…(Ten reviews were excluded from the original sample of 300 reviews, either because they were COVID-19 reviews, or the online publication date could not be determined.) This sample was derived for a separate project [14] and used stricter inclusion criteriasystematic reviews had to have clearly stated objective(s), report the sources searched, include an assessment of risk of bias, and contain at least one pairwise meta-analysis. For each review, one author (SM) extracted the date of search (including the format and where reported) and date the review was published online.…”

Section: Data Search Study Selection and Data Extractionmentioning

confidence: 99%

Are COVID-19 systematic reviews up to date and can we tell? A cross-sectional study

McDonald

Turner

Nguyen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundCOVID-19 led to a rapid acceleration in the number of systematic reviews. Readers need to know how up to date evidence is when selecting reviews to inform decisions. This cross-sectional study aimed to evaluate how easily the currency of COVID-19 systematic reviews published early in the pandemic could be determined and how up to date these reviews were at the time of publication.MethodsWe searched for systematic reviews and meta-analyses relevant to COVID-19 added to PubMed in July 2020 and January 2021, including any that were first published as preprints. We extracted data on the date of search, number of included studies, and date first published online. For the search date, we noted the format of the date and where in the review this was reported. A sample of non-COVID-19 systematic reviews from November 2020 served as a comparator.ResultsWe identified 246 systematic reviews on COVID-19. In the abstract of these reviews, around half (48%) reported the complete search date (day/month/year) and 43% failed to report any date. In the full text, the complete date and no date were reported in 82% and 6% of reviews, respectively. The median time from last search to publication online was 91 days (IQR 63–130). Time from search to publication was similar for the subset of 15 rapid or living reviews (92 days) but shorter for the 29 reviews published as preprints (37 days). The median number of studies or publications included per review was 23 (IQR 12–40). In the sample of 290 non-COVID SRs, around a quarter (27%) reported the complete search date and a third (34%) did not include any date in the abstract. The median time from search to publication online was 253 days (IQR 153–381) and each review included a median of 12 studies (IQR 8–21).ConclusionsDespite the context of the pandemic and the need to easily ascertain the currency of systematic reviews, reporting of the search date information for COVID-19 reviews was suboptimal. Reviews were conducted and published more quickly than non-COVID reviews but potentially at the expense of adequate reporting.

show abstract

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Cited by 8 publications

References 59 publications

Systematic reviewers' perspectives on sharing review data, analytic code, and other materials: A survey

Systematic reviewers' perspectives on sharing review data, analytic code, and other materials: A survey

Guidance to Best Tools and Practices for Systematic Reviews

Are COVID-19 systematic reviews up to date and can we tell? A cross-sectional study

Contact Info

Product

Resources

About