Methodological quality of systematic reviews on Chinese herbal medicine: a methodological survey

Abstract

Background Systematic reviews (SRs) synthesise the best evidence of effectiveness and safety on Chinese herbal medicine (CHM). Decision-making should be supported by the high-quality evidence of prudently conducted SRs, but the trustworthiness of conclusions may be limited by poor methodological rigour. Methods This survey aimed to examine the methodological quality of a representative sample of SRs on CHM published during January 2018 to March 2020. We conducted literature search in Cochrane Database of Systematic Reviews, MEDLINE via Ovid, and EMBASE via Ovid. Eligible SRs must be in Chinese or English with at least one meta-analysis on the treatment effect of any CHM documented in the 2015 Chinese Pharmacopoeia. Two reviewers extracted the bibliographical characteristics of SRs and appraised their methodological quality using AMSTAR 2 (Assessing the Methodological Quality of Systematic Reviews 2). The associations between bibliographical characteristics and methodological quality were investigated using Kruskal-Wallis tests and Spearman’s rank correlation coefficients. Results We sampled and appraised one hundred forty-eight SRs. Overall, one (0.7%) was of high methodological quality; zero (0%), four (2.7%), and one-hundred forty-three (96.6%) SRs were of moderate, low, and critically-low quality. Only thirteen SRs (8.8%) provided a pre-defined protocol; none (0%) provided justifications for including particular primary study designs; six (4.1%) conducted a comprehensive literature search; two (1.4%) provided a list of excluded studies; nine (6.1%) undertook meta-analysis with appropriate methods; and seven (4.7%) reported funding sources of included primary studies. Cochrane reviews had higher overall quality than non-Cochrane reviews (P < 0.001). SRs with European funding support were less likely to have critically-low quality when compared with their counterparts (P = 0.020). SRs conducted by more authors (rs = 0.23; P = 0.006) and published in higher impact factor journals (rs = 0.20; P = 0.044) were associated with higher methodological quality. Conclusions Our results indicated that the methodological quality of SRs on CHM is low. Future authors should enhance the methodological quality through registering a priori protocols, justifying selection of study designs, conducting comprehensive literature search, providing a list of excluded studies with rationales, using appropriate method for meta-analyses, and reporting funding sources among primary studies.

What Are the Determinants of the Quality of Systematic Reviews in the International Journals of Occupational Medicine? A Methodological Study Review of Published Literature

“…An evident relationship between AMSTAR-2 score and the study impact factor score was found, and this is in agreement with the results found by Cheung et al (Chinese Medicine) [27], McGregor et al (pain management) [28], and in disagreement with Pauletto et al (dentistry) [26] and Chow et al (gynecology and obstetrics) [30]. In our study, an increase in the impact factor was associated with a higher score of the AMSTAR-2, suggesting that occupational medicine journals with a higher impact factor are more likely to publish systematic reviews with a higher methodological quality, as suggested by Saha et al, who found that usually researchers consider impact factor as a reasonable indicator of study quality [36].…”

“…Moreover, our study also found a significant and positive association between the AMSTAR-2 score and the number of authors in the study, and again, this is in agreement with the results found by Cheung et al (Chinese Medicine) [ 27 ] and in disagreement with Chow et al [ 30 ]. These findings could be linked to the so-called phenomenon of “knowledge diffusion”.…”

“…Cheung et al assessed the methodological quality of systematic reviews on Chinese herbal medicine, finding higher levels in SRs conducted by more authors and published in higher impact factor journals [ 27 ].…”

Integrative oncology in cancer care – implementation factors: mixed-methods systematic review

“…First, although the primary studies that were included contained the perspectives of many key stakeholders in IO service implementation, such as BMDs, T&CMPs and nurses, a large proportion of the samples in the studies comprised patients and BMDs, while a significantly smaller proportion comprised caregivers, administrators and pharmacists (as shown in table 1). Therefore, future primary studies should focus on the views of these groups, especially pharmacists, as their role in implementing IO services may be crucial when herbal or dietary supplements are provided, given the limited amount of information on these supplements 51. In fact, BMDs and nurses are often concerned about possible adverse effects from the use of herbal supplements or due to herb–drug interactions, so are hesitant about using T&CM 12.…”

Quality of systematic reviews with meta‐analyses of resveratrol: A methodological systematic review

“…For instance, Fleming et al (2014) evaluated 327 systematic reviews of medical interventions and concluded that those published in higher IF journals were significantly associated with higher methodological score obtained from AMSTAR, which is the older version of the AMSTAR‐2 tool. Cheung et al (2022) investigated the methodological quality of 148 systematic reviews on herbal medicines and reported significant positive associations between IF, number of authors, and methodological quality. Overall, the MA's authors should pay more attention to the methodological flaws identified by our research.…”