Transparency and Reproducibility of Observational Cohort Studies Using Large Healthcare Databases.

Abstract

The scientific community and decision-makers are increasingly concerned about transparency and reproducibility of epidemiologic studies using longitudinal healthcare databases. We explored the extent to which published pharmacoepidemiologic studies using commercially available databases could be reproduced by other investigators. We identified a nonsystematic sample of 38 descriptive or comparative safety/effectiveness cohort studies. Seven studies were excluded from reproduction, five because of violation of fundamental design principles, and two because of grossly inadequate reporting. In the remaining studies, >1,000 patient characteristics and measures of association were reproduced with a high degree of accuracy (median differences between original and reproduction <2% and <0.1). An essential component of transparent and reproducible research with healthcare databases is more complete reporting of study implementation. Once reproducibility is achieved, the conversation can be elevated to assess whether suboptimal design choices led to avoidable bias and whether findings are replicable in other data sources.

Fournier's gangrene in patients with type 2 diabetes using second‐line antidiabetic medications

“…PSs were estimated using logistic regression models that included the following baseline patient characteristics: age on index, gender, urinary tract infections (UTIs), diseases of the genitourinary system, hospitalization for bacterial infections, prior use of immune‐modulating or ‐suppressing drugs, HIV/AIDS, cancer, obesity, cardiovascular disease (CVD), microvascular disease, alcohol abuse, liver and renal disease, and chronic obstructive pulmonary disease (COPD). Analysis was performed using the Aetion Evidence Platform version 3.13 …”

Effects of expanding the look‐back period to all available data in the assessment of covariates

“…All studies of interest were selected from a non‐systematic sample of peer‐reviewed database studies, published between 2008 and 2014. Wang et al assessed the reproducibility of these studies in Aetion using the same Optum/United data source and study design to match the reported results …”

“…Each study with the exception of angioedema events in association with use of drugs that act on the renin–angiotensin–aldosterone system included the basic confounders with deciles of the propensity score and deciles of the trimmed propensity score in their assessment (Table ). Basing our comparison across approaches on the results presented in Wang et al , we did not have any basis of comparison for these two assessments . We looked for marked differences in the treatment effect estimates across the fixed and expanded look‐back approaches, as well as meaningful increases in the prevalences of covariates captured.…”

“…We thank Dr. Shirley Wang at Brigham and Women's Hospital (DoPE) for her published work on reproducibility of observational studies in Aetion, which served as the initial framework for this project. We thank the staff at Ludwig Maximilian University of Munich, Department of Medical Informatics, Biostatistics and Epidemiology (IBE), who provided insight and expertise that assisted the research.…”

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0

“…As others have noted, reporting the specific codes used to define these measures is critical for transparency and reproducibility 47,63 especially in databases where there can be substantial ambiguity in code choice.…”