Empirical evidence for outcome reporting bias in randomized clinical trials of acupuncture: comparison of registered records and subsequent publications

Abstract

BackgroundOutcome reporting bias has received widespread recognition and been considered to pose two threats to the validity of clinical decision making because they overestimate the effect of treatments or distort the results of trials. However, the problem of outcome-reporting bias has not been systematically studied among randomized clinical trials of acupuncture. Our objectives were to evaluate the consistency between the registered records and subsequent publications with respect to outcomes and other data as well as to determine whether outcome-reporting bias favors significant primary outcomes.MethodsA systematic search of 15 registries was conducted from their inception to January 2014 to identify randomized clinical trials on acupuncture for which the status was listed as ‘completed.’ The subsequent publications were retrieved by searching PubMed and three Chinese databases. Basic characteristics and the registration information were extracted from the registered records and publications. We performed comparisons regarding primary outcomes and other data between the registered records and subsequent publications to assess the consistency and selective outcome reporting.ResultsEighty-eight trials on acupuncture with 96 published reports were identified. Only 19.3% (17/88) were registered before the start of the trial, suggesting prospective registration. The trial registration number was unavailable in 36 published reports (37.5%). A comparison of registered and published primary outcomes could be conducted in 71 publications (74.0%), and the inconsistency of the primary outcomes was identified in 45.1% (32 of 71); 71.4% (15 of 21) had a discrepancy that favored statistically significant primary outcomes, while 28.6% (6 of 21) favored nonsignificant primary outcomes. Furthermore, the other inconsistencies between the registry records and subsequent publications involved the inclusion criteria (54.7%), exclusion criteria (47.9%) and controls (22.9%).ConclusionsWe find that prospective registration for randomized clinical trials on acupuncture is insufficient, selective outcome reporting is prevalent, and the change of primary outcomes is intended to favor statistical significance. These discrepancies in outcome reporting may lead to biased and misleading results of randomized clinical trials on acupuncture. To ensure publication of reliable and unbiased results, further promotion and implementation of trial registration are still needed.Electronic supplementary materialThe online version of this article (doi:10.1186/s13063-014-0545-5) contains supplementary material, which is available to authorized users.

A systematic review of comparisons between protocols or registrations and full reports in primary biomedical research

“…Full reports: from January 2013 to October 2014 40/76 RCTs Outcome reporting Not reported Not reported No funding Rosati, 2016, Trials [ 39 ] Pediatrics Clinical trial registries including the United States National Institute of Health (clinicaltrials.gov), the ISRCTN registry, the Nederlands Trials Register (NTR), the Australian and New Zealand Clinical Trial Registry (ACTRN), and the Clinical Trial Registry-India (CTRI) 20 consecutive RCTs published from July to November 2013 in the journal Pediatrics Registrations: from July 2013 to November 2013. Full reports: from July 2013 to November 2013 20/20 RCTs Outcome reporting, sample size, statistical plan, result interpretation Most conducted in USA Not reported No funding received Su, 2015, Trials [ 43 ] Acupuncture 15 major international trial registries PubMed and three Chinese databases Registrations and full reports: up to January 2014 88/96 RCT Outcome reporting, inclusion criteria, and comparator Most conducted in western countries (74%) Sample size ranged from 10 to 960 Not reported Turner, 2012, PLoS Medicine [ 44 ] Antipsychotic Trials FDA reviews on eight second-generation antipsychotic drugs for the treatment of schizophrenia PubMed and Cochrane Central Register of Controlled Trials FDA reviews: from December 1993 to May 2005. Full reports: up to May 2010 24/20 Clinical trials Outcome reporting, result presentation Not reported Not reported Academic funding Systematic review (n = 4) Mhaskar, 2012, Journal of Clinical Epidemiology [ 30 ] Oncology Protocols of trials conducted by eight National Cancer Institute Cooperative Groups Publications of trials conducted by eight National Cancer Institute Cooperative Groups Protocols and full reports: Up to 2006 429/429 RCTs (Phase III) Study sample, comparator, outcome, study design, statistical plan and randomization Not reported Not reported Academic funding You, 2012, Journal of clinical oncology [ ...…”

Need for Registration and Reporting of Acupuncture Trials in Parkinson’s Disease in Korea

“…Thus, it is very likely that trials are still not being registered and results are not being reported. Studies have also shown the potential for reporting bias in acupuncture trials and publications [ 25 ]. Furthermore, assessment of registration information for acupuncture trials has reinforced the need for methodological design improvements [ 26 , 27 ].…”

Major discrepancies between what clinical trial registries record and paediatric randomised controlled trials publish

“…The discrepancies we identified in declaring funding and conflict of interests in nine trials, in the number of eligible and enrolled participants in eight clinical trials and in another nine inclusion and exclusion criteria not being respected, emphasise the generally imperfect reporting. These major discrepancies, especially those involving changes in the original study hypotheses, trial designs, study conduction and reporting outcomes raise concern on trustworthiness in scientific trials, as previous papers have underlined [ 5 – 9 , 13 , 15 , 16 ].…”

“…In the 1920s and 1930s two scientists in different research fields [ 2 , 3 ] helped enormously to reappraise statistical theory and methodology in designing trials, thus clarifying how bias influences research. Previous papers have already compared protocols and registered data for clinical outcomes in published RCTs in various clinical settings [ 4 – 7 ], and three studies have investigated discrepancies in selectively reported outcomes [ 8 , 9 ]. Even though concealment and blinding tools can control human factors in RCT designs [ 10 ] and the International Committee of Medical Journal Editors (ICMJE) [ 11 ] and Consolidated Standards of Reporting Trials (CONSORT) [ 12 ] receive wide consensus, a recent survey among journal editors endorsing the ICMJE and CONSORT established policies disclosed that only 27 % of the 33 respondents cross-checked the data reported in the submitted manuscript against the data registered prospectively in clinical trial registries (CTRs) [ 13 ].…”