Measuring the effect of reviewers on manuscript change: A study on a sample of submissions to Royal Society journals (2006–2017)

Bianchi, Federico; García-Costa, Daniel; Grimaldo, Francisco; Squazzoni, Flaminio

doi:10.1016/j.joi.2022.101316

Cited by 8 publications

(2 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evidence from the literature supports the interpretation that citations added during review are less likely to be substantive, leaving early stage references with a proportionately higher level of these. This is because biomedical articles do not change much during the review process, leaving little scope for this stage of biomedical science communication to effect change ( 48 – 52 ). Over 80% of bioRxiv authors self-report posting preprints before receiving reviews ( 50 ).…”

Section: Discussionmentioning

confidence: 99%

Predicting substantive biomedical citations without full text

Hoppe

Arabi

Hutchins

2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Insights from biomedical citation networks can be used to identify promising avenues for accelerating research and its downstream bench-to-bedside translation. Citation analysis generally assumes that each citation documents substantive knowledge transfer that informed the conception, design, or execution of the main experiments. Citations may exist for other reasons. In this paper, we take advantage of late-stage citations added during peer review because these are less likely to represent substantive knowledge flow. Using a large, comprehensive feature set of open access data, we train a predictive model to identify late-stage citations. The model relies only on the title, abstract, and citations to previous articles but not the full-text or future citations patterns, making it suitable for publications as soon as they are released, or those behind a paywall (the vast majority). We find that high prediction scores identify late-stage citations that were likely added during the peer review process as well as those more likely to be rhetorical, such as journal self-citations added during review. Our model conversely gives low prediction scores to early-stage citations and citation classes that are known to represent substantive knowledge transfer. Using this model, we find that US federally funded biomedical research publications represent 30% of the predicted early-stage (and more likely to be substantive) knowledge transfer from basic studies to clinical research, even though these comprise only 10% of the literature. This is a threefold overrepresentation in this important type of knowledge flow.

show abstract

Section: Discussionmentioning

confidence: 99%

Predicting substantive biomedical citations without full text

Hoppe

Arabi

Hutchins

2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Unfortunately, there is little understanding of how this developmental function actually works [13][14][15][16]. While research on specific journals has shown that exposure to different rounds of peer review could increase the quality of manuscripts-including later submissions to other journals if rejected [17], other studies have suggested that reviewers are keen to preferably concentrate on theoretical aspects rather than rigour, methodology and statistical content [11,18].…”

Section: Introductionmentioning

confidence: 99%

Does peer review improve the statistical content of manuscripts? A study on 27 467 submissions to four journals

et al. 2022

Self Cite

View full text Add to dashboard Cite

Improving the methodological rigour and the quality of data analysis in manuscripts submitted to journals is key to ensure the validity of scientific claims. However, there is scant knowledge of how manuscripts change throughout the review process in academic journals. Here, we examined 27 467 manuscripts submitted to four journals from the Royal Society (2006–2017) and analysed the effect of peer review on the amount of statistical content of manuscripts, i.e. one of the most important aspects to assess the methodological rigour of manuscripts. We found that manuscripts with both initial low or high levels of statistical content increased their statistical content during peer review. The availability of guidelines on statistics in the review forms of journals was associated with an initial similarity of statistical content of manuscripts but did not have any relevant implications on manuscript change during peer review. We found that when reports were more concentrated on statistical content, there was a higher probability that these manuscripts were eventually rejected by editors.

show abstract