Evaluation of Reproducible and Transparent Research Practices in Sports Medicine Research: A Cross-sectional study

Abstract

Disclosures:The authors report no conflicts of interest. Word count: 2,978Author contributions: MV and DT conceived the study and designed the protocol. MV and DT supervised the study. MV received research funding. SE and IF extracted data. SE and IF conducted data analysis and organization. MA conducted the final data analysis. SE, IF, MA, DT, and MV drafted the manuscript. All authors provided critical feedback, ideas, and editing for the manuscript and have approved the final version. SE assumes responsibility for this manuscript. Abstract: Background:In recent years, urgency has been placed on the "reproducibility crisis" facing biomedical research. Despite efforts toward improvement, certain elements needed to reproduce a study are often lacking from publications. The current state of reproducibility within the sports medicine research community remains unknown.Purpose: Our study sought to evaluate the presence of eight indicators of reproducibility and transparency to determine the current state of research reporting in sports medicine research. Study Design: Cross-sectional reviewMethods: Using the National Library of Medicine catalog, we identified 41 MEDLINE-indexed, English language sports medicine journals. From the 41 journals, we randomly sampled 300 publications that were recorded on PubMed as being published between January 1, 2014, and December 31, 2018. Two investigators extracted data in duplicate and blinded fashion. Results:Of the 300 publications sampled, 280 were accessible and were screened for empirical data.Studies that lack empirical data were excluded from our analysis. Of the remaining 195 with empirical data, 10 (5.13%) publications provided data availability statements, 1 (0.51%) provided a protocol, 0 (0.0%) provided an analysis script, and 9 (4.62%) were pre registered.Conclusion: Reproducibility and transparency indicators are lacking in sports medicine publications. The majority of publications lack the necessary resources for reproducibility such as material, data, analysis scripts, or protocol availability. While the current state of reproducibility cannot be fixed overnight, we feel combined efforts of data sharing, open access, and verifying disclosure statements can help to improve overall reporting.

How often do cancer researchers make their data and code available and what factors are associated with sharing?

“…Specifically, we noted that only one of the 59 articles that declared data was available complied with our FAIR assessment. This observation, depending on how availability for reuse is defined, is unfortunately consistent with this body of research which has reported 50-100% reductions in availability following interrogation of sharing statements [15][16][17][18][19][20][21][22][23][24][25] ; with factors such as the lack of unique and permanent identifiers, meta-data and licensing terms being noted as major pitfalls. 30,47 Furthermore, while we also noted a strong relationship between mandatory data sharing policies and actual data availability, we unfortunately also observed similarly sub-optimal compliance with these policies too; a finding that has been noted by other studies both inside and outside of medicine.…”

“…50 In contrast to the growth of data sharing declarations over time, despite claims that code sharing is becoming increasingly normalised across many scientific fields 53 , we note persistently low code sharing rates in medicine since 2014. [15][16][17][18][19][20][21][22] Furthermore, none of the six studies in our sample that were subject to mandatory code sharing policies reported code to be available. A finding which is consistent with the only other study to the authors' knowledge that has examined compliance with code sharing policies in medicine by Grayling and Wheeler (2020) 28 who reported that only 18% of the 91 methodological articles describing novel adaptive clinical trial designs that were subject to mandatory sharing policies made their code available.…”

“…Rather our observations that 19% and 4% of cancer researchers declared data and code were publicly available respectively are consistent with several studies reporting low, but increasing, declaration rates ranging between 3-24% and 0-2% respectively over several medical disciplines between 2014-2018. [15][16][17][18][19][20][21][22][23][24][25] The increase of declarations over time -particularly data availability declarations -is likely due to the growing number of medical journals that are adopting stronger policies on data and code sharing, particularly those that are requiring the addition of availability statements. For example, we note that a quarter of the unique 235 journals analysed in our study had adopted a mandatory data sharing policy for some or all data, which is higher than a previous survey of medical journal editors in the previous year.…”

“…First, our sample size was 4-10 times bigger than the annual estimates of previous research evaluating data and code sharing rates in medicine between 2014 and 2018. [15][16][17][18][19][20][21][22] Random sampling of articles allowed us to make inferences about sharing rates more broadly than studies focused on sharing rates for articles published in specific journals (e.g., high impact journals), or using certain study designs (e.g., randomised controlled trials). Second, we examined entire articles and supplementary materials for declarations of data and code sharing.…”

“…14 While significant progress has been made towards increasing the availability of the products of research (i.e., data, code and materials), previous research on the topic has reported low availability of data (0-6%) and code (0-2%) across many medical fields between 2014 and 2018. [15][16][17][18][19][20][21][22][23][24][25] Furthermore, other research in medicine has shown sub-optimal compliance with journal data and code sharing policies. [26][27][28] In this study we build on previous research to investigate how frequently cancer researchers share the data underlying their research, as well as the code used to perform statistical analyses in a large random sample of published cancer studies.…”