Nature of Our Literature

Abstract

Scientists rely upon an accurate scientific literature in order to build and test new theories about the natural world. In the past decade, observational studies of the scientific literature have indicated that numerous questionable research practices and poor reporting practices may be hindering scientific progress. In particular, 3 recent studies have indicated an implausibly high rate of studies with positive (i.e., hypothesis confirming) results. In sports medicine, a field closely related to kinesiology, studies that tested a hypothesis indicated support for their primary hypothesis ~70% of the time. However, a study of journals that cover the entire field of kinesiology has yet to be completed, and the quality of other reporting practices, such as clinical trial registration, has not been evaluated. In this study we retrospectively evaluated 300 original research articles from the flagship journals of North America (Medicine and Science in Sports and Exercise), Europe (European Journal of Sport Science), and Australia (Journal of Science and Medicine in Sport). The hypothesis testing rate (~64%) and positive result rate (~81%) were much lower than what has been reported in other fields (e.g., psychology), and there was only weak evidence for our hypothesis that the positive result rate exceeded 80%. However, the positive result rate is still considered unreasonably high. Additionally, most studies did not report trial registration, and rarely included accessible data indicating rather poor reporting practices. The majority of studies relied upon significance testing (~92%), but it was more concerning that a majority of studies (~82%) without a stated hypothesis still relied upon significance testing. Overall, the positive result rate in kinesiology is unacceptably high, despite being lower than other fields such as psychology, and most published manuscripts demonstrated subpar reporting practices

Proposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science

“…Also, this method never allows us to gather evidence in favour of the null hypothesis; only evidence against it [46]. Along with the high positive result rate in the field [33], this protocol focuses on statistically significant effects for reasons of feasibility. An attempt to replicate a non-significant effect (where p > 0.05) would require infeasible sample sizes more than 16 times the original to obtain 80% replication power [41].…”

“…Studies involving Bayesian analyses are excluded. Our focus on statistically significant key variables (e.g., counter-movement jump height) stems from the findings of Twomey and colleagues [ 33 ]. They reported an 80% positive results rate in sports and exercise science, i.e., most researchers in the field are reporting statistically significant findings.…”

Reporting and transparent research practices in sports medicine and orthopaedic clinical trials: a meta-research study

“…A recent study in kinesiology shows even lower rates of pre-registration, data availability statements and data sharing in open repositories. 31 Sports medicine researchers have already noted that preregistration and registered reports can prevent questionable research practices 26 (table 1) or make them easier to detect. 32 Data were not shared in public repositories, suggesting that this topic requires special attention.…”

Raising the bar in sports performance research

“…Along with Open Science practices, we strongly encourage authors to use reporting guidelines appropriate for the research design, such as STROBE for observational designs (Von Elm et al, 2007) and CONSORT for intervention studies (Moher et al, 2010). Although more than a decade has passed since sports performance researchers were encouraged to use reporting guidelines (Atkinson et al, 2008), unfortunately, few studies in our discipline explicitly state using them (Twomey et al, 2021). Nevertheless, evidence suggests that the use of reporting guidelines improves reporting and study quality (Cobo et al, 2011;Plint et al, 2006;Turner et al, 2012).…”