Universal Principled Review: A Community-Driven Method to Improve Peer Review

Krummel, Matthew F.; Blish, Catherine A.; Kuhns, Michael S.; Cadwell, Ken; Oberst, Andrew; Goldrath, Ananda W.; Ansel, K. Mark; Chi, Hongbo; O’Connell, Ryan M.; Wherry, E. John; Pepper, Marion; Brodsky, Igor E.; Chang, John T.; Arron, Joseph R.; Haining, Nick; Bhattacharya, Deepta; Anderson, Mark; Rothlin, Carla V.; Schwab, Susan R.; Belkaid, Yasmine; Molofsky, Ari B.; Savage, Pete; Mucida, Daniel; Iwasaki, Akiko; Victora, Gabriel D.; Hamerman, Jessica A.; Masopust, David; Barton, Greg; Kaech, Susan M.; Woodruff, Prescott G.; Stetson, Daniel B.; Scharschmidt, Tiffany C.; Kedl, Ross M.; Zúñiga, Elina I.; Hoffmann, Alexander; Williams, Matt; Mayer‐Barber, Katrin D.; Shin, Sunny; Bensinger, Steven J.; Lu, Li-Fan; Looney, Mark R.; Round, J; Amigorena, Sébastian; Yewdell, Jonathan W.; Sun, Joseph C.; Harty, John T.

doi:10.1016/j.cell.2019.11.029

Cited by 11 publications

(6 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Enlarging the pool of reviewers to potentially an entire scientific community and accelerating the whole process requires a standard for peer reviews [32]: for example some aspects might be taken over by AI assistants (such as the Artificial Intelligence Review Assistant (AIRA) [33] leaving to the reviewers the sole task of evaluating the content of a paper. Building smart contracts for peer reviews might accelerate this novel process of standardization.…”

Section: A New Community-driven Standard?mentioning

confidence: 99%

Ants-Review: a Protocol for Incentivized Open Peer-Reviews on Ethereum

Trovò,

Massari

2021

Preprint

View full text Add to dashboard Cite

Peer-review is a necessary and essential quality control step for scientific publications but lacks proper incentives. Indeed, the process, which is very costly in terms of time and intellectual investment, not only is not remunerated by the journals but is also not openly recognized by the academic community as a relevant scientific output for a researcher. Therefore, scientific dissemination is affected in timeliness, quality and fairness. Here, to solve this issue, we propose a blockchain-based incentive system that rewards scientists for peer-reviewing other scientists' work and that builds up trust and reputation. We designed a privacy-oriented protocol of smart contracts called Ants-Review that allows authors to issue a bounty for open anonymous peer-reviews on Ethereum. If requirements are met, peer reviews will be accepted and paid by the approver proportionally to their assessed quality. To promote ethical behaviour and inclusiveness the system implements a gamified mechanism that allows the whole community to evaluate the peer-reviews and vote for the best ones.

show abstract

Section: A New Community-driven Standard?mentioning

confidence: 99%

Ants-Review: a Protocol for Incentivized Open Peer-Reviews on Ethereum

Trovò,

Massari

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This is difficult; some authors use independent peer review (e.g. Jarwal et al (2009)) but of course peer review itself is a fraught metric (see Brezis and Birukou 2020;Krummel et al 2019). We investigate test of time awards in the "An analysis of quantitative publication metrics" section; in some fields, retractions may also provide a useful signal.…”

Section: Quantitative Metrics As Proxy Indicatorsmentioning

confidence: 99%

What is meaningful research and how should we measure it?

Helmer

Blumenthal

Paschen³

2020

Scientometrics

View full text Add to dashboard Cite

We discuss the trend towards using quantitative metrics for evaluating research. We claim that, rather than promoting meaningful research, purely metric-based research evaluation schemes potentially lead to a dystopian academic reality, leaving no space for creativity and intellectual initiative. After sketching what the future could look like if quantitative metrics are allowed to proliferate, we provide a more detailed discussion on why research is so difficult to evaluate and outline approaches for avoiding such a situation. In particular, we characterize meaningful research as an essentially contested concept and argue that quantitative metrics should always be accompanied by operationalized instructions for their proper use and continuously evaluated via feedback loops. Additionally, we analyze a dataset containing information about computer science publications and their citation history and indicate how quantitative metrics could potentially be calibrated via alternative evaluation methods such as test of time awards. Finally, we argue that, instead of over-relying on indicators, research environments should primarily be based on trust and personal responsibility.

show abstract

“…Were we really protecting science by delaying a yes/no decision, or by rejecting solid but circumscribed work? This experience has built on previous attempts to limit reviewer demands and multiple resubmissions (Malhotra and Marder, 2015) and on a push to develop a reviewer compact that encourages our peers generally to approach manuscripts with the intention of seeing good data published (Krummel et al 2019). With support and even explicit guidance from journal editors, many of us requested additions more parsimoniously.…”

Section: Refining Publication Practices Is Possible and Desirablementioning

confidence: 99%

Lessons of COVID-19: A roadmap for post-pandemic science

Freedman

Headley

Serwas

et al. 2020

Journal of Experimental Medicine

View full text Add to dashboard Cite

The response to the COVID-19 crisis across most research institutions mandated ceasing nonessential research activities in order to minimize the spread of the virus in our communities. With minimal notice, experiments were terminated, cell lines were frozen, mouse colonies were culled, and trainees were prevented from performing bench research. Still, despite the interruption of experimental productivity, the shutdown has proven for many PIs and trainees that doing and thinking science are not activities that are bound to the laboratory. Furthermore, the shutdowns have solidified important emerging trends and forced us to further innovate to get the most out of working remotely. We hope that some of these innovations, hard-gained in this difficult time, will persist and develop into new paradigms—lessons that will improve our science and our relationship to the climate and community beyond the current pandemic.

show abstract

Universal Principled Review: A Community-Driven Method to Improve Peer Review

Cited by 11 publications

References 3 publications

Ants-Review: a Protocol for Incentivized Open Peer-Reviews on Ethereum

Ants-Review: a Protocol for Incentivized Open Peer-Reviews on Ethereum

What is meaningful research and how should we measure it?

Lessons of COVID-19: A roadmap for post-pandemic science

Contact Info

Product

Resources

About