Methodological assessment of systematic reviews and meta‐analyses on COVID‐19: A meta‐epidemiological study

Abstract

Rationale, aims, and objectives COVID‐19 has caused an ongoing public health crisis. Many systematic reviews and meta‐analyses have been performed to synthesize evidence for better understanding this new disease. However, some concerns have been raised about rapid COVID‐19 research. This meta‐epidemiological study aims to methodologically assess the current systematic reviews and meta‐analyses on COVID‐19. Methods We searched in various databases for systematic reviews with meta‐analyses published between 1 January 2020 and 31 October 2020. We extracted their basic characteristics, data analyses, evidence appraisal, and assessment of publication bias and heterogeneity. Results We identified 295 systematic reviews on COVID‐19. The median time from submission to acceptance was 33 days. Among these systematic reviews, 73.9% evaluated clinical manifestations or comorbidities of COVID‐19. Stata was the most used software programme (43.39%). The odds ratio was the most used effect measure (34.24%). Moreover, 28.14% of the systematic reviews did not present evidence appraisal. Among those reporting the risk of bias results, 14.64% of studies had a high risk of bias. Egger's test was the most used method for assessing publication bias (38.31%), while 38.66% of the systematic reviews did not assess publication bias. The I 2 statistic was widely used for assessing heterogeneity (92.20%); many meta‐analyses had high values of I 2 . Among the meta‐analyses using the random‐effects model, 75.82% did not report the methods for model implementation; among those meta‐analyses reporting implementation methods, the DerSimonian‐Laird method was the most used one. Conclusions The current systematic reviews and meta‐analyses on COVID‐19 might suffer from low transparency, high heterogeneity, and suboptimal statistical methods. It is recommended that future systematic reviews on COVID‐19 strictly follow well‐developed guidelines. Sensitivity analyses may be performed to examine how the synthesized evidence might depend on different methods for appraising evidence, assessing publication bias, and implementing meta‐analysis models.

Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency

“…Although the median time from search to publication was nine weeks, this ranged from a few days to 39 weeks. Studies have reported substantially shorter acceptance times by journals for COVID-19 publications [ 4 , 68 ], but this has not been sufficient to prevent half the remdesivir reviews being out of date when published. The prominence of preprints during the pandemic has accelerated scientific dissemination [69] , but although most of the reviews published as preprints in our analysis were posted within three weeks of the search date, this was no guarantee of currency, with 45% (4/9) still out of date at time of publication.…”

Good Statistical Practices for Contemporary Meta-Analysis: Examples Based on a Systematic Review on COVID-19 in Pregnancy

“…However, it is difficult to apply these metrics to clinical practice for future research. Over the last decade, much effort has been made to promote the reporting of the prediction interval (PI) in a meta-analysis, but only a small proportion of metaanalyses adopt this recommendation in the current literature [9,[59][60][61][62][63]. The PI represents the expected range of the true effects in future studies, making it easier to apply metaanalysis results to clinical practice.…”

“…The reliability of the synthesized evidence depends critically on appropriate methods used to perform meta-analyses [3,4]. However, despite the mass production of meta-analyses, it has been found that many meta-analyses need improvements in their methodological quality [5][6][7][8][9][10]. This is a particularly crucial issue in the COVID-19 pandemic because of the concerns about the expedited peer-review process [11][12][13][14].…”

“…Various methods with potentially better performance are available and can be readily implemented with various statistical software programs [25][26][27][28][29]. Nevertheless, the DerSimonian-Laird (DL) method continues to dominate contemporary meta-analyses [9]. Some popular software programs for meta-analysis (e.g., Review Manager) use the DL method as the default and perhaps the only option.…”

Patient-important outcomes reported in randomized controlled trials of pharmacologic treatments for COVID-19: a protocol of a META-epidemiological study

“…Researchers are rushing to test pharmacological interventions that may decrease the personal and social burden of the disease as well as producing associated guideline recommendations [ 10 ]. Current literature, however, points to some methodological shortcuts trialist have taken such as low transparency, high heterogeneity, and suboptimal statistical methods [ 11 ]. Trials reportedly have small sample size are single center, and present with redundancy in research questions [ 9 ].…”