Can Orthopaedics become the Gold Standard for Reproducibility? A Roadmap to Success

Abstract

Background:Scientific research is replete with poor accessibility to data, materials, and protocol, which limits the reproducibility of a study. Transparency with regard to materials, protocols, and raw data sets enhances reproducibility by providing the critical information necessary to verify, replicate, and resynthesize research findings. The extent to which transparency and reproducibility exist in the field of orthopaedics is unclear. In our study, we aimed to evaluate transparency and reproducibility-related characteristics of randomly sampled publications in orthopaedic journals. Methods:We used the National Library of Medicine catalog to identify English language and MEDLINE-indexed orthopaedic journals. From the 74 journals meeting our inclusion criteria, we randomly sampled 300 publications using a refined PubMed search that were published between January 1, 2014, and December 31, 2018. Two investigators were trained for data extraction and analysis. Both investigators were blinded and independently extracted data from the 300 studies. Results:Our initial search yielded 68,102 publications, from which we drew a random sample of 300 publications.Of these 300 publications, 286 were screened for empirical data and 14 were inaccessible. For analysis purposes, we excluded publications without empirical data. Of the 182 with empirical data, 13 studies (7.1%) included a data availability statement, 9 (4.9%) reported materials were available, none (0.0%) provided analysis scripts, 2 (1.1%) provided access to the protocol used, 5 (2.7%) were preregistered, and only 2 (1.1%) provided a statement about being a replicated study. Conclusions:Components necessary for reproducibility are lacking in orthopaedic surgery journals. The vast majority of publications did not provide data or material availability statements, protocols, or analysis scripts, and had no preregistration statements. Intervention is needed to improve reproducibility in the field of orthopaedics. The current state of reproducibility in orthopaedic surgery could be improved by combined efforts from funding agencies, authors, peer reviewers, and journals alike. Level of Evidence: N/A

How often do cancer researchers make their data and code available and what factors are associated with sharing?

“…Specifically, we noted that only one of the 59 articles that declared data was available complied with our FAIR assessment. This observation, depending on how availability for reuse is defined, is unfortunately consistent with this body of research which has reported 50-100% reductions in availability following interrogation of sharing statements [15][16][17][18][19][20][21][22][23][24][25] ; with factors such as the lack of unique and permanent identifiers, meta-data and licensing terms being noted as major pitfalls. 30,47 Furthermore, while we also noted a strong relationship between mandatory data sharing policies and actual data availability, we unfortunately also observed similarly sub-optimal compliance with these policies too; a finding that has been noted by other studies both inside and outside of medicine.…”

“…50 In contrast to the growth of data sharing declarations over time, despite claims that code sharing is becoming increasingly normalised across many scientific fields 53 , we note persistently low code sharing rates in medicine since 2014. [15][16][17][18][19][20][21][22] Furthermore, none of the six studies in our sample that were subject to mandatory code sharing policies reported code to be available. A finding which is consistent with the only other study to the authors' knowledge that has examined compliance with code sharing policies in medicine by Grayling and Wheeler (2020) 28 who reported that only 18% of the 91 methodological articles describing novel adaptive clinical trial designs that were subject to mandatory sharing policies made their code available.…”

“…First, our sample size was 4-10 times bigger than the annual estimates of previous research evaluating data and code sharing rates in medicine between 2014 and 2018. [15][16][17][18][19][20][21][22] Random sampling of articles allowed us to make inferences about sharing rates more broadly than studies focused on sharing rates for articles published in specific journals (e.g., high impact journals), or using certain study designs (e.g., randomised controlled trials). Second, we examined entire articles and supplementary materials for declarations of data and code sharing.…”

Rates and predictors of data and code sharing in the medical and health sciences: Protocol for a systematic review and individual participant data meta-analysis.

“…14 To date, numerous studies have investigated how prevalent data and code sharing is. With regard to medicine and health, this research has reported traditionally low, but increasing rates of sharing and use of data availability statements across many fields, including but not limited to: biomedicine, [15][16][17][18] cardiology, 19 oncology, 20 orthopaedics, 21 otolaryngology, 22 radiology 23 and COVID-19-related research. 24 Previous research has also highlighted low sharing of clinical trial data both publicly, 25 as well as in response to reasonable private requests (e.g.…”

Rates and predictors of data and code sharing in the medical and health sciences: Protocol for a systematic review and individual participant data meta-analysis.

“…14 To date, numerous studies have investigated how prevalent data and code sharing is. With regard to medicine and health, this research has reported traditionally low, but increasing rates of sharing and use of data availability statements across many fields, including but not limited to: biomedicine, [15][16][17][18] cardiology, 19 oncology, 20 orthopaedics, 21 otolaryngology, 22 radiology 23 and COVID-19-related research. 24 Previous research has also highlighted low sharing of clinical trial data both publicly, 25 as well as in response to reasonable private requests (e.g.…”