A descriptive analysis of the data availability statements accompanying medRxiv preprints and a comparison with their published counterparts

Abstract

Objective To determine whether medRxiv data availability statements describe open or closed data—that is, whether the data used in the study is openly available without restriction—and to examine if this changes on publication based on journal data-sharing policy. Additionally, to examine whether data availability statements are sufficient to capture code availability declarations. Design Observational study, following a pre-registered protocol, of preprints posted on the medRxiv repository between 25th June 2019 and 1st May 2020 and their published counterparts. Main outcome measures Distribution of preprinted data availability statements across nine categories, determined by a prespecified classification system. Change in the percentage of data availability statements describing open data between the preprinted and published versions of the same record, stratified by journal sharing policy. Number of code availability declarations reported in the full-text preprint which were not captured in the corresponding data availability statement. Results 3938 medRxiv preprints with an applicable data availability statement were included in our sample, of which 911 (23.1%) were categorized as describing open data. 379 (9.6%) preprints were subsequently published, and of these published articles, only 155 contained an applicable data availability statement. Similar to the preprint stage, a minority (59 (38.1%)) of these published data availability statements described open data. Of the 151 records eligible for the comparison between preprinted and published stages, 57 (37.7%) were published in journals which mandated open data sharing. Data availability statements more frequently described open data on publication when the journal mandated data sharing (open at preprint: 33.3%, open at publication: 61.4%) compared to when the journal did not mandate data sharing (open at preprint: 20.2%, open at publication: 22.3%). Conclusion Requiring that authors submit a data availability statement is a good first step, but is insufficient to ensure data availability. Strict editorial policies that mandate data sharing (where appropriate) as a condition of publication appear to be effective in making research data available. We would strongly encourage all journal editors to examine whether their data availability policies are sufficiently stringent and consistently enforced.

Reproducibility of COVID-19 pre-prints

“…Notably, 28% of sampled pre-prints from bioRxiv contained open data markers and 22% of sampled arXiv pre-prints contained markers of open code, the highest proportions of any repository for each type of marker. Our results are similar to McGuinness , Sheppard ( 2021 ), who focus on medRxiv and find that 23% describe open data.…”

“…They find that data availability was maintained for most of their sample, varying by journal data sharing policy with greater improvements in openness among manuscripts published in journals mandating data sharing. While limited to medRxiv, the results of McGuinness and Sheppard ( 2021 ) align with our own work and provide initial evidence to suggest that data availability is generally maintained or improved between the pre-print and publication stages.…”

“…There is limited literature examining the relationship between data and code availability in manuscripts between the pre-print and publication stages. McGuinness and Sheppard ( 2021 ) examine differences in data availability statements between medRxiv pre-prints and their published counterparts. They find that data availability was maintained for most of their sample, varying by journal data sharing policy with greater improvements in openness among manuscripts published in journals mandating data sharing.…”

“…The current system is biased against those who follow best practice. McGuinness and Sheppard ( 2021 ) advocate for ‘(s)trict editorial policies that mandate data sharing,’ and other changed norms are needed.…”

Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals

“…The rate of subsequent publications of preprints cited in the four world-leading medical journals is higher than the rate of publication of preprints reported in other studies, ranging from 6.9 to 21.1% [1,4,5,7,[19][20][21]. However, the rates of publication reported in those studies are probably underestimated since they relied on information about subsequent publication in MedRxiv, which we observed to be largely inaccurate, confirming previous reports [9].…”