Evaluation of Reproducibility in Urology Publications

Rauh, Shelby; Johnson, Bradley S.; Bowers, Aaron; Tritz, Daniel; Vassar, Matt

doi:10.1101/773945

Cited by 3 publications

(13 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, we noted that only one of the 59 articles that declared data was available complied with our FAIR assessment. This observation, depending on how availability for reuse is defined, is unfortunately consistent with this body of research which has reported 50-100% reductions in availability following interrogation of sharing statements [15][16][17][18][19][20][21][22][23][24][25] ; with factors such as the lack of unique and permanent identifiers, meta-data and licensing terms being noted as major pitfalls. 30,47 Furthermore, while we also noted a strong relationship between mandatory data sharing policies and actual data availability, we unfortunately also observed similarly sub-optimal compliance with these policies too; a finding that has been noted by other studies both inside and outside of medicine.…”

Section: Discussionsupporting

confidence: 77%

“…Rather our observations that 19% and 4% of cancer researchers declared data and code were publicly available respectively are consistent with several studies reporting low, but increasing, declaration rates ranging between 3-24% and 0-2% respectively over several medical disciplines between 2014-2018. [15][16][17][18][19][20][21][22][23][24][25] The increase of declarations over time -particularly data availability declarations -is likely due to the growing number of medical journals that are adopting stronger policies on data and code sharing, particularly those that are requiring the addition of availability statements. For example, we note that a quarter of the unique 235 journals analysed in our study had adopted a mandatory data sharing policy for some or all data, which is higher than a previous survey of medical journal editors in the previous year.…”

Section: Discussionmentioning

confidence: 99%

“…14 While significant progress has been made towards increasing the availability of the products of research (i.e., data, code and materials), previous research on the topic has reported low availability of data (0-6%) and code (0-2%) across many medical fields between 2014 and 2018. [15][16][17][18][19][20][21][22][23][24][25] Furthermore, other research in medicine has shown sub-optimal compliance with journal data and code sharing policies. [26][27][28] In this study we build on previous research to investigate how frequently cancer researchers share the data underlying their research, as well as the code used to perform statistical analyses in a large random sample of published cancer studies.…”

mentioning

confidence: 99%

See 2 more Smart Citations

How often do cancer researchers make their data and code available and what factors are associated with sharing?

Hamilton

Page

Finch

et al. 2022

Preprint

View full text Add to dashboard Cite

Background: Various stakeholders are calling for increased availability of data and code from cancer research. However, it is unclear how commonly these products are shared, and what factors are associated with sharing. Our objective was to evaluate how frequently oncology researchers make data and code available, and explore factors associated with sharing. Methods: A cross-sectional analysis of a random sample of 306 articles indexed in PubMed in 2019 presenting original cancer research was performed. Outcomes of interest included the prevalence of affirmative sharing declarations and the rate with which declarations connected to useable data. We also investigated associations between sharing rates and several journal characteristics (e.g., sharing policies, publication models), study characteristics (e.g., cancer rarity, study design), open science practices (e.g., pre-registration, preprinting) and citation rates between 2020-2021. Results: One in five studies declared data were publicly available (95% CI: 15-24%). However, when actual data availability was investigated this percentage dropped to 16% (95% CI: 12-20%), and then to less than 1% (95% CI: 0-2%) when data were checked for compliance with key FAIR principles. While only 4% of articles that used inferential statistics reported code to be available (10/274, 95% CI: 2-6%), the odds of reporting code to be available were 5.6 times higher for researchers who shared data. Compliance with mandatory data and code sharing policies was observed in 48% and 0% of articles, respectively. However, 88% of articles included data availability statements when required. Policies that encouraged data sharing did not appear to be any more effective than not having a policy at all. The only factors associated with higher rates of data sharing were studying rare cancers and using publicly available data to complement original research. Conclusions: Data and code sharing in oncology occurs infrequently, and at a lower frequency than would be expected due to non-compliance with journal policies. There is also a large gap between those declaring data to be available, and those archiving data in a way that facilitates its reuse. We encourage journals to actively check compliance with sharing policies, and researchers consult community accepted guidelines when archiving the products of their research.

show abstract

Section: Discussionsupporting

confidence: 77%

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

How often do cancer researchers make their data and code available and what factors are associated with sharing?

Hamilton

Page

Finch

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These findings are consistent with prior reports of reproducibility and transparency in the medical ,10,12,14 and surgical literature. 7,8,11 Table 5 presents a comparison of the current study's results juxtaposed with results obtained from similar methods across other surgical disciplines and associated comparison tests for multiple proportions. In terms of open access status, plastic surgery articles were much less available-11.3 percent of journals were available by means of open access-than articles from other disciplines.…”

Section: Resultsmentioning

confidence: 99%

“…These findings are consistent with prior reports of reproducibility and transparency in the medical ,10,12,14 and surgical literature. 7,8,11…”

Section: Discussionmentioning

confidence: 99%

Reproducible Research Practices in the Plastic Surgery Literature

Ascha

Katabi

Stevens

et al. 2022

Plastic &Amp; Reconstructive Surgery

View full text Add to dashboard Cite

Background: Reproducible research—defined as the ability to replicate a study with its published materials and procedures—is integral to ensuring the validity of published studies and promoting scientific advancement. The primary aim of this study was to quantitatively evaluate the reproducibility and transparency of research in the plastic surgery literature. Methods: A cross-sectional study was performed. Articles published in 12 plastic surgery journals over a 5-year period were randomly selected. Reproducibility-related and transparency-related variables were blindly and independently collected by two reviewers using previously published methods. Descriptive statistics and univariate analysis were performed for outcomes of interest. Results: The initial search yielded 18,461 publications, from which 400 were randomly selected. A total of 397 publications met inclusion criteria, of which 203 were empirical studies eligible for analysis of reproducibility-related and transparency-related variables. Among the empirical studies, most did not have a data availability statement (97.0 percent; 95 percent CI, 93.7 to 98.9). Only seven (3.4 percent; 95 percent CI, 1.4 to 7.0) were linked to an accessible protocol, four (2.0 percent; 95 percent CI, 0.5 to 5.0) were preregistered, and no studies provided analysis scripts or claimed to replicate another study. Of the 202 studies evaluated for material availability, only 17 (8.4 percent; 95 percent CI, 5.0 to 13.1) had a material availability statement. Conclusions: There is an evident lack of reproducible research practices in plastic surgery literature. The majority of plastic surgery publications do not provide information and raw materials necessary to reproduce empirical studies. Increasing awareness at the individual and institutional levels can improve research quality and transparency.

show abstract

How often do cancer researchers make their data and code available and what factors are associated with sharing?

Hamilton

Page

Finch

et al. 2022

BMC Med

View full text Add to dashboard Cite

Background Various stakeholders are calling for increased availability of data and code from cancer research. However, it is unclear how commonly these products are shared, and what factors are associated with sharing. Our objective was to evaluate how frequently oncology researchers make data and code available and explore factors associated with sharing. Methods A cross-sectional analysis of a random sample of 306 cancer-related articles indexed in PubMed in 2019 which studied research subjects with a cancer diagnosis was performed. All articles were independently screened for eligibility by two authors. Outcomes of interest included the prevalence of affirmative sharing declarations and the rate with which declarations connected to data complying with key FAIR principles (e.g. posted to a recognised repository, assigned an identifier, data license outlined, non-proprietary formatting). We also investigated associations between sharing rates and several journal characteristics (e.g. sharing policies, publication models), study characteristics (e.g. cancer rarity, study design), open science practices (e.g. pre-registration, pre-printing) and subsequent citation rates between 2020 and 2021. Results One in five studies declared data were publicly available (59/306, 19%, 95% CI: 15–24%). However, when data availability was investigated this percentage dropped to 16% (49/306, 95% CI: 12–20%), and then to less than 1% (1/306, 95% CI: 0–2%) when data were checked for compliance with key FAIR principles. While only 4% of articles that used inferential statistics reported code to be available (10/274, 95% CI: 2–6%), the odds of reporting code to be available were 5.6 times higher for researchers who shared data. Compliance with mandatory data and code sharing policies was observed in 48% (14/29) and 0% (0/6) of articles, respectively. However, 88% of articles (45/51) included data availability statements when required. Policies that encouraged data sharing did not appear to be any more effective than not having a policy at all. The only factors associated with higher rates of data sharing were studying rare cancers and using publicly available data to complement original research. Conclusions Data and code sharing in oncology occurs infrequently, and at a lower rate than would be expected given the prevalence of mandatory sharing policies. There is also a large gap between those declaring data to be available, and those archiving data in a way that facilitates its reuse. We encourage journals to actively check compliance with sharing policies, and researchers consult community-accepted guidelines when archiving the products of their research.

show abstract

Evaluation of Reproducibility in Urology Publications

Cited by 3 publications

References 38 publications

How often do cancer researchers make their data and code available and what factors are associated with sharing?

How often do cancer researchers make their data and code available and what factors are associated with sharing?

Reproducible Research Practices in the Plastic Surgery Literature

How often do cancer researchers make their data and code available and what factors are associated with sharing?

Contact Info

Product

Resources

About