Open Up: A Survey on Open and Non-anonymized Peer Reviewing

Abstract

We present a discussion and analysis regarding the benefits and limitations of open and non-anonymized peer review based on literature results and responses to a survey on the reviewing process of alt.chi, a more or less open-review track within the CHI conference, the predominant conference in the field of human-computer interaction (HCI). This track currently is the only implementation of an open-peer-review process in the field of HCI while, with the recent increase in interest in open science practices, open review is now being considered and used in other fields. We collected 30 responses from alt.chi authors and reviewers and found that, while the benefits are quite clear and the system is generally well liked by alt.chi participants, they are reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.

Comparison of early career researchers and senior career researchers as peer reviewers: A questionnaire survey in China

“…Additionally, if the review comments of ECRs are consistent with those of editors and most other reviewers, they will feel gladder and more comfortable than SCRs. In this regard, to ensure the independence of the review, journals should attempt to prevent reviewers (especially ECR reviewers) from seeing the opinions of other reviewers and editors before they submit their reviews (Besançon et al, 2019), and it may be helpful to give the reviewers some feedback on their comments.…”

Challenges for the Academic Editor in the Scientific Publication

“…The impact on applied science Preprint Publication without prior peer review Social impact Online references, e.g. on social networks Mega-journals Covers a very large number of topics Decolonisation De-centring scientific quality from the Anglophilic centre Source: Own production, based on Tennant et al, 2019;Herteliu et al, 2017;Baffy et al, 2020;Gammelgaard, 2016;Hughes, Stone, Aravopoulou, Tiu Wright, & Machtynger, 2018;Ferreira & Serpa, 2018a;Besançon, Rönnberg, Löwgren, Tennant, & Cooper, 2019;Xu, 2018;Lemke, Mehrazar, Mazarakis, & Peters, 2019;Wakeling et al, 2017.…”

“…In short, academic editors, in their activity, will have to be able to cope in a world of increasing open access dominance, with challenges, such as (i) journal indexing and metrics, measured through the number of citations per article (Tennant et al, 2019;Herteliu et al, 2017;Baffy et al, 2020;Gammelgaard, 2016); (ii) the increasing pressure for articles to explicitly indicate their "practitioner impact" (Hughes et al, 2018, p. 2); (iii) the growing preprint publication with manuscripts not previously peer-reviewed before being made publicly available (Ferreira & Serpa, 2018a;Besançon et al, 2019;Tennant et al, 2019); (iv) the presence of references of articles in social networks assessed through Altmetrics or similar indicators (Xu, 2018;Lemke et al, 2019); (v) the mega-journals, which have a focus that covers a very large number of topics, such as, for example, humanities and/or social sciences (Wakeling et al, 2017); and, finally, (vi) decolonise the international scientific publication, acknowledging that scientific quality is not present only in the Anglophone centre's model and language (Banks et al, 2018;Trahar, Juntrasook, Burford, von Kotze, & Wildemeersch, 2019). Within this context, there may be a temptation to manipulate the ranking results through excessive and unnecessary citation of a given journal's own publications (Rovira, Codina, Guerrero-Solé, & Lopezosa, 2019;Herteliu et al, 2017), for example, through "coercive citation, review articles, editorials and letters, and online queuing (i.e., the number of articles pre-posted on the web)" (Wilhite, Fong, & Wilhite, 2019, p. 1514.…”

The benefits to climate science of including early-career scientists as reviewers

“…Here, we focus on a report produced by the Intergovernmental Panel on Climate Change (IPCC) that provides an assessment of the scientific, technical and socio-economic literature on the current state of knowledge on climate change (IPCC, 2013a). Ensuring effective climate change adaptation and mitigation requires policymakers to be informed by the scientific community through robust and evidence-based reports reflecting the scientific consensus (Bolin, 2007;Tollefson, 2010;Ding et al, 2011;Lewandowsky et al, 2013). To best achieve this goal, the scientific community requires the inclusion of scientists from heterogeneous backgrounds and experiences (Maibach et al, 2014;Hallegatte et al, 2016), including ECSs, who can provide diverse perspectives.…”