Average effect estimates remain similar as evidence evolves from single trials to high-quality bodies of evidence: a meta-epidemiologic study

Gartlehner, Gerald; Dobrescu, Andreea; Evans, Tammeka Swinson; Thaler, Kylie; Nußbaumer, Barbara; Sommer, Isolde; Lohr, Kathleen N

doi:10.1016/j.jclinepi.2015.02.013

Cited by 7 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One ME study [ 29 ] demonstrated that overall trials showed significantly much lower treatment effect estimates than that of first trial (ratio of effect size: 2.67, 95% CI: 2.12–3.37), although the remaining ME study [ 30 ] did not find such association (ratio of effect size: 1.03, 95% CI: 0.98–1.08). Several other trial-level characteristics including sufficient follow-up, placebo control and statistician involvement, among others have been investigated as well, with no significant associations being found (Additional file 10 : Appendix 10).…”

Section: Resultsmentioning

confidence: 99%

“…A tentative explanation for the differences between these subgroups is that trials on complementary medicine had a higher probability of suffering from methodological flaws [ 31 ]; iii) larger treatment effect estimates in first trial as compared with subsequent trial were consistently observed, regardless of the trial size, risk of bias or effect size of the first trial for continuous outcomes, indicating the robustness of the association. However, such explorations are missing in binary outcomes, although inconsistencies were observed between the two available ME studies [ 29 , 30 ]. That invites future ME studies to address.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Trial-level characteristics associate with treatment effect estimates: a systematic review of meta-epidemiological studies

Wang¹,

Song²,

Lin³

et al. 2022

BMC Med Res Methodol

View full text Add to dashboard Cite

Background To summarize the up-to-date empirical evidence on trial-level characteristics of randomized controlled trials associated with treatment effect estimates. Methods A systematic review searched three databases up to August 2020. Meta-epidemiological (ME) studies of randomized controlled trials on intervention effect were eligible. We assessed the methodological quality of ME studies using a self-developed criterion. Associations between treatment effect estimates and trial-level characteristics were presented using forest plots. Results Eighty ME studies were included, with 25/80 (31%) being published after 2015. Less than one-third ME studies critically appraised the included studies (26/80, 33%), published a protocol (23/80, 29%), and provided a list of excluded studies with justifications (12/80, 15%). Trials with high or unclear (versus low) risk of bias on sequence generation (3/14 for binary outcome and 1/6 for continuous outcome), allocation concealment (11/18 and 1/6), double blinding (5/15 and 2/4) and smaller sample size (4/5 and 2/2) significantly associated with larger treatment effect estimates. Associations between high or unclear risk of bias on allocation concealment (5/6 for binary outcome and 1/3 for continuous outcome), double blinding (4/5 and 1/3) and larger treatment effect estimates were more frequently observed for subjective outcomes. The associations between treatment effect estimates and non-blinding of outcome assessors were removed in trials using multiple observers to reach consensus for both binary and continuous outcomes. Some trial characteristics in the Cochrane risk-of-bias (RoB2) tool have not been covered by the included ME studies, including using validated method for outcome measures and selection of the reported results from multiple outcome measures or multiple analysis based on results (e.g., significance of the results). Conclusions Consistently significant associations between larger treatment effect estimates and high or unclear risk of bias on sequence generation, allocation concealment, double blinding and smaller sample size were found. The impact of allocation concealment and double blinding were more consistent for subjective outcomes. The methodological and reporting quality of included ME studies were dissatisfactory. Future ME studies should follow the corresponding reporting guideline. Specific guidelines for conducting and critically appraising ME studies are needed.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Trial-level characteristics associate with treatment effect estimates: a systematic review of meta-epidemiological studies

Wang¹,

Song²,

Lin³

et al. 2022

BMC Med Res Methodol

View full text Add to dashboard Cite

show abstract

“…When we detected differences in the effect estimates or conclusions between the preprint and peerreviewed article, two investigators independently classified these changes. We used the typology developed by Gartlehner et al (30) to classify these changes but had to adapt it because of the range of effect estimates we identified. We considered the statistical significance of the primary outcome between the preprint and peer-reviewed article as having changed when at least one of the two effect estimates had a P-value that was deemed statistically significant in either the preprint or publication, and not statistically significant in the other.…”

Section: Data Extraction and Analysismentioning

confidence: 99%

Full Publication of Preprint Articles in Prevention Research: An Analysis of Publication Proportions and Results Consistency

Sommer

Sunder-Plaßmann

Ratajczak

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Introduction There is concern that preprint articles will lead to an increase in the amount of scientifically invalid work available. The objectives of this study were 1) to determine the proportion of prevention preprints published within 12 months, 2) to assess the consistency of the effect estimates and conclusions between preprint and published articles, and 3) to explore the reasons for the nonpublication of preprints. Methods We developed a web crawler to search the preprint server medRxiv for prevention studies posted from January 1 to September 30, 2020. An update search was conducted 12 months later. We dually screened the results for prevention articles and developed a scheme to classify changes in effect sizes and conclusions. We modeled the effect of a set of predictors on the proportion of preprints published in peer-reviewed journals. We also developed a questionnaire for authors of unpublished preprints. Results Of the 329 prevention preprints that met our eligibility criteria, almost half (48.9%) were published in a peer-reviewed journal within 12 months of being posted, with the median time being 5.3 months (range 0.1–11.3 months). While 27 out of 161 (16.8%) published preprints showed some change in the magnitude of the primary outcome effect estimate, 4.4% were classified as having a major change. The style or wording of the conclusion changed in 42.2%, while the content of the conclusion changed in 3.1%. Preprints on chemoprevention, with a cross-sectional design, and with public and noncommercial funding had the highest probabilities of publication. The main reasons for the nonpublication of preprints were journal rejection or lack of time. Conclusion The reliability of preprint articles for evidence-based decision-making is questionable. Less than half of the preprint articles on prevention research are published in a peer-reviewed journal within 12 months, and significant changes in effect sizes and/or conclusions are still possible during the peer-review process.

show abstract

“…Reporting methods used to select harms 10) Report the methods used to prioritize harms, differentiate serious from frequent but less serious harms, and indicate interventions for which serious harms are not believed to be an issue and why.…”

Section: Unanticipated Harmsmentioning

confidence: 99%

“…The assessment and reporting of harms is often suboptimal, [1][2][3] studies are often too short to evaluate important long-term harms and have inadequate statistical power to evaluate serious but uncommon harms, 5,6 patients enrolled in research studies are frequently at lower risk for harms than those encountered in clinical practice, 7 potentially resulting in underestimation of harms, and important data on harms may be unpublished or selectively reported. [9][10][11] In 2005, AHRQ funded a series of white papers on challenges in evidence synthesis that included an article on evaluation of harms. 5 It highlighted unique challenges in finding and selecting data on harms, rating the quality of harms reporting, and synthesizing and displaying from studies reporting harms.…”

Section: Introductionmentioning

confidence: 99%

Prioritization and Selection of Harms for Inclusion in Systematic Reviews

Chou¹,

Baker²,

Bañez³

et al. 2017

View full text Add to dashboard Cite

None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.Persons using assistive technology may not be able to fully access information in this report. For assistance, contact EffectiveHealthCare@ahrq.hhs.gov PrefaceThe Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public-and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies and strategies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.To improve the scientific rigor of these evidence reports, AHRQ supports empiric research by the EPCs to help understand or improve complex methodologic issues in systematic reviews. These methods research projects are intended to contribute to the research base in and be used to improve the science of systematic reviews. They are not intended to be guidance to the EPC program, although they may be considered by EPCs along with other scientific research when determining EPC program methods guidance.AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers and the health care system as a whole by providing important information to help improve health care quality. The reports undergo peer review prior to their release as a final report.

show abstract

Average effect estimates remain similar as evidence evolves from single trials to high-quality bodies of evidence: a meta-epidemiologic study

Cited by 7 publications

References 12 publications

Trial-level characteristics associate with treatment effect estimates: a systematic review of meta-epidemiological studies

Trial-level characteristics associate with treatment effect estimates: a systematic review of meta-epidemiological studies

Full Publication of Preprint Articles in Prevention Research: An Analysis of Publication Proportions and Results Consistency

Prioritization and Selection of Harms for Inclusion in Systematic Reviews

Contact Info

Product

Resources

About