RipetaScore: Measuring the Quality, Transparency, and Trustworthiness of a Scientific Work

Abstract

A wide array of existing metrics quantifies a scientific paper's prominence or the author's prestige. Many who use these metrics make assumptions that higher citation counts or more public attention must indicate more reliable, better quality science. While current metrics offer valuable insight into scientific publications, they are an inadequate proxy for measuring the quality, transparency, and trustworthiness of published research. Three essential elements to establishing trust in a work include: trust in the paper, trust in the author, and trust in the data. To address these elements in a systematic and automated way, we propose the ripetaScore as a direct measurement of a paper's research practices, professionalism, and reproducibility. Using a sample of our current corpus of academic papers, we demonstrate the ripetaScore's efficacy in determining the quality, transparency, and trustworthiness of an academic work. In this paper, we aim to provide a metric to evaluate scientific reporting quality in terms of transparency and trustworthiness of the research, professionalism, and reproducibility.

Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality

“…This means that the task of spreading and checking best practices is difficult. Checklists can help guide best practices, and enforcing these checklists should lead to improved reporting standards [ 41 ], but given the scale of publishing, the use of automatic checklist tools such as SciScore and others, more focused tools such as Barzooka (continuous data in bar graphs), JetFighter (color-blind accessibility in visualizations), ODDPub (data and code availability), and RipetaScore (authorship, ethics, and data or code availability) [ 42 - 45 ], should help authors and reviewers improve manuscripts and address common checklist items and omissions consistently across many journals. In addition, automatic checklist completion can only help speed up the review process, which is a notoriously slow endeavor [ 46 ].…”

In defense of quantitative metrics in researcher assessments

“…Bibliometrics should encompass indicators of best research practices (e.g., frequency of data sharing, code sharing, protocol registration, and replications) as a free, publicly available resource covering all the open-access literature [ 12 ]. Examples include PLOS’s Open Science Indicators , which are currently capturing data sharing in repositories, code sharing, and preprint posting, and the Dimensions Research Integrity product and its proposed Ripeta Score [ 13 ]. Centralized open-access assessments can also scrutinize for elements of poor research practices (e.g., signs of image manipulation) at a massive scale.…”