Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts

Abstract

The primary aims of peer review are to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts’ content to be variable, but generally modest and positive. I qualitatively analysed 1430 peer reviewers’ comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers’ comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers’ comments. Reviewers’ comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.

Does peer review improve the statistical content of manuscripts? A study on 27 467 submissions to four journals

“…However, this type of research only rarely integrates full data on manuscripts during each stage of the editorial process and data on review reports, at the same time covering different journals [ 38 , 39 ]. Integrating data on manuscripts and reports is key to providing a context-specific picture of peer review and editorial processes, not to mention the possibility of assessing changes and revisions of manuscripts due to peer review [ 28 , 40 ]. Although difficult, pooling across-journal data is instrumental to examine the emergence of peer review practices that are shared in various communities [ 24 , 35 ].…”

“…In our opinion, this was an appropriate design strategy considering the type of journals and areas of research in our dataset and the fact that statistics is a standardized field. However, integrating our measurements with qualitative analysis of the text by human experts would be a significant step forward [ 40 ]. This would also help to assess the potentially negative effect of reviewer requests on manuscript change as well as inform us about the link between increased statistical content and methodological quality and rigour of reported studies.…”

“…This implies that any measurement is only explorative and caution must be used when drawing any conclusions from a study’s findings. On the one hand, even research on manuscript change in preprint–publication pairs estimates the potential effects of peer review only indirectly as the link between manuscripts and reports is missing [ 40 , 47 , 48 ]. On the other hand, research on the content of peer review reports from available report repositories, e.g.…”

Publishing in Hospitality and Tourism Journals

“…For editors, it helps reduce bias and ensure quality through multiple rounds of peerreview. For journals, it creates a hierarchy in the publishing system, favouring those with a rigorous PRP over those without (Sabaj-Meruane et al, 2016;Stephen, 2022). The PRP serves as a self-regulating mechanism to ensure the quality of the publishing system (Horbach & Halffman, 2018).…”

“…The review criteria generally assess the suitability of the study design for the research question, the appropriate selection of the sample and data collection techniques, the suitability of the data analysis techniques, and the straightforward interpretation of the results. Studies from various disciplines (Beatty et al, 1992;Beyer, 1978;Black et al, 1998;Garcia-Costa, Forte et al, 2022;Lent et al, 2015;Stephen, 2022) often analyze reviewers' judgments on the relative importance of these criteria. It is noteworthy, however, that there are few such studies in tourism scholarship.…”