Reproducibility of published meta-analyses on clinical psychological interventions

Abstract

Meta-analysis is one of the most useful and powerful research approaches, however, this relevance relies on its credibility. In recent years different concerns about the credibility of psychological research have emerged. Analytical reproducibility of scientific results could be considered as the minimal threshold of it. In this study, our purpose was to assess the analytical reproducibility and data reusability of a set of published meta-analyses. From a random sample of 100 papers containing at least one meta-analysis on the effectiveness of interventions in psychology used in a previous study, 217 meta-analyses were selected. We first tried to retrieve the original data by recovering a data file, recoding the data from document files (pdf, doc), or on request. Second, through a multi-stage workflow, we tried to reproduce the main results of each meta-analysis using these data. The original data were retrieved for 146 meta-analyses from different sources. Of these, in the first stage 52 showed a discrepancy larger than 5% in the main results, in 25 of them, this discrepancy was solved with minor adjustments, or correction of coding errors. In the remaining 27, different issues were identified in an in-depth review of the papers, such as reporting inconsistencies, lack of some data, or transcription errors. Current practices of data sharing in meta-analyses hamper the reusability of meta-analytic data. On the other hand, the implementation of new tools would help to avoid certain errors in the meta-analysis reporting process.