The devil is in the details: Reporting and transparent research practices in sports medicine and orthopedic clinical trials

Schulz, Robert; Langen, Georg; Prill, Robert; Cassel, Michael; Weissgerber, Tracey L.

doi:10.1101/2021.07.20.21260565

“…Approximately two thirds of papers did not contain enough information to determine whether observations were excluded without explanation [ 28 ]. A small exploratory analysis suggested that clinical trials that included flow charts were more likely to fully report attrition than studies that provided this information in the text [ 29 ].…”

Section: Discussionmentioning

confidence: 99%

Replacing bar graphs of continuous data with more informative graphics: are we making progress?

Riedel

¹

,

Schulz

²

,

Kazezian

³

et al. 2022

Self Cite

View full text Add to dashboard Cite

Recent work has raised awareness about the need to replace bar graphs of continuous data with informative graphs showing the data distribution. The impact of these efforts is not known. This observational meta-research study examined how often scientists in different fields use various graph types, and assessed whether visualization practices have changed between 2010 and 2020. We developed and validated an automated screening tool, designed to identify bar graphs of counts or proportions, bar graphs of continuous data, bar graphs with dot plots, dot plots, box plots, violin plots, histograms, pie charts, and flow charts. Papers from 23 fields (approximately 1,000 papers/field/year) were randomly selected from PubMed Central and screened (n=227,998). F1 scores for different graphs ranged between 0.83 and 0.95 in the internal validation set. While the tool also performed well in external validation sets, F1 scores were lower for uncommon graphs. Bar graphs are more often used incorrectly to display continuous data than they are used correctly to display counts or proportions. The proportion of papers that use bar graphs of continuous data varies markedly across fields (range in 2020: 4%-58%), with high rates in biochemistry and cell biology, complementary and alternative medicine, physiology, genetics, oncology and carcinogenesis, pharmacology, microbiology and immunology. Visualization practices have improved in some fields in recent years. Fewer than 25% of papers use flow charts, which provide information about attrition and the risk of bias. This study highlights the need for continued interventions to improve visualization and identifies fields that would benefit most.

show abstract

“…The study was pre-registered on the Open Science Framework (RRID:SCR_003238) and all generated data were made openly available. 22 Additional details regarding sample selection and screening, data abstraction, a sample size calculation and data for each included study can be found in the online supplemental materials .…”

Section: Methodsmentioning

confidence: 99%

Reporting and transparent research practices in sports medicine and orthopaedic clinical trials: a meta-research study

Schulz

¹

,

Langen²,

Prill³

et al. 2022

Self Cite

View full text Add to dashboard Cite

ObjectivesTransparent reporting of clinical trials is essential to assess the risk of bias and translate research findings into clinical practice. While existing studies have shown that deficiencies are common, detailed empirical and field-specific data are scarce. Therefore, this study aimed to examine current clinical trial reporting and transparent research practices in sports medicine and orthopaedics.SettingExploratory meta-research study on reporting quality and transparent research practices in orthopaedics and sports medicine clinical trials.ParticipantsThe sample included clinical trials published in the top 25% of sports medicine and orthopaedics journals over 9 months.Primary and secondary outcome measuresTwo independent reviewers assessed pre-registration, open data and criteria related to scientific rigour, like randomisation, blinding, and sample size calculations, as well as the study sample, and data analysis.ResultsThe sample included 163 clinical trials from 27 journals. While the majority of trials mentioned rigour criteria, essential details were often missing. Sixty per cent (95% confidence interval (CI) 53% to 68%) of trials reported sample size calculations, but only 32% (95% CI 25% to 39%) justified the expected effect size. Few trials indicated the blinding status of all main stakeholders (4%; 95% CI 1% to 7%). Only 18% (95% CI 12% to 24%) included information on randomisation type, method and concealed allocation. Most trials reported participants’ sex/gender (95%; 95% CI 92% to 98%) and information on inclusion and exclusion criteria (78%; 95% CI 72% to 84%). Only 20% (95% CI 14% to 26%) of trials were pre-registered. No trials deposited data in open repositories.ConclusionsThese results will aid the sports medicine and orthopaedics community in developing tailored interventions to improve reporting. While authors typically mention blinding, randomisation and other factors, essential details are often missing. Greater acceptance of open science practices, like pre-registration and open data, is needed. As these practices have been widely encouraged, we discuss systemic interventions that may improve clinical trial reporting.

show abstract

Replacing bar graphs of continuous data with more informative graphics: Are we making progress?

Riedel

¹

,

Schulz

²

,

Kazezian

³

et al. 2022

Preprint

Self Cite

3

0

View full text Add to dashboard Cite

Recent work has raised awareness about the need to replace bar graphs of continuous data with informative graphs showing the data distribution. The impact of these efforts is not known. This observational meta-research study examined how often scientists in different fields use various graph types, and assessed whether visualization practices have changed between 2010 and 2020. We developed and validated an automated screening tool, designed to identify bar graphs of counts or proportions, bar graphs of continuous data, bar graphs with dot plots, dot plots, box plots, violin plots, histograms, pie charts, and flow charts. Papers from 23 fields (approximately 1,000 papers/field/year) were randomly selected from PubMed Central and screened (n=227,998). F1 scores for different graphs ranged between 0.83 and 0.95 in the internal validation set. While the tool also performed well in external validation sets, F1 scores were lower for uncommon graphs. Bar graphs are more often used incorrectly to display continuous data than they are used correctly to display counts or proportions. The proportion of papers that use bar graphs of continuous data varies markedly across fields (range in 2020: 4%-58%), with high rates in biochemistry and cell biology, complementary and alternative medicine, physiology, genetics, oncology and carcinogenesis, pharmacology, microbiology and immunology. Visualization practices have improved in some fields in recent years. Fewer than 25% of papers use flow charts, which provide information about attrition and the risk of bias. This study highlights the need for continued interventions to improve visualization and identifies fields that would benefit most.

show abstract

The devil is in the details: Reporting and transparent research practices in sports medicine and orthopedic clinical trials

Cited by 4 publications

References 80 publications

Replacing bar graphs of continuous data with more informative graphics: are we making progress?

Replacing bar graphs of continuous data with more informative graphics: are we making progress?

Reporting and transparent research practices in sports medicine and orthopaedic clinical trials: a meta-research study

Replacing bar graphs of continuous data with more informative graphics: Are we making progress?

Contact Info

Product

Resources

About