Refinement: Measuring informativeness of ratings in the absence of a gold standard

Grant, Sheridan; Meilă, Marina; Erosheva, Elena A.; Lee, Carole

doi:10.1111/bmsp.12268

Cited by 2 publications

(1 citation statement)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The evaluative method described here (collecting both ratings and partial rankings) and the combination of these streams of information into one integrated score provides a coherent way to assess a group of research proposals. While funding schemas like partial lotteries embrace the seemingly stochastic nature of grant review scores and outcomes ([ 51 ]), our approach conversely works to decrease the noise in review outcomes by extracting more meaningful information from reviewers ([ 52 ]). Moreover, the output of this process is a priority list for project funding that is structured optimally for the decision type of many secondary, programmatic funding panels and offers a confidence interval or a probability surrounding a given project’s priority.…”

Section: Discussionmentioning

confidence: 99%

A new approach to grant review assessments: score, then rank

Gallo¹,

Pearce²,

Lee³

et al. 2023

Res Integr Peer Rev

View full text Add to dashboard Cite

Background In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications. Methods We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical “toy” examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges’ evaluations. Results For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions. Conclusions A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.

show abstract

Section: Discussionmentioning

confidence: 99%

A new approach to grant review assessments: score, then rank

Gallo¹,

Pearce²,

Lee³

et al. 2023

Res Integr Peer Rev

View full text Add to dashboard Cite

show abstract

Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter‐rater reliability

Bartoš,

Martinková

2024

Brit J Math & Statis

View full text Add to dashboard Cite

Inter‐rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the IRR and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss other possible uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health‐related measurement, and implement the computations in the R package IRR2FPR.

show abstract

Refinement: Measuring informativeness of ratings in the absence of a gold standard

Cited by 2 publications

References 49 publications

A new approach to grant review assessments: score, then rank

A new approach to grant review assessments: score, then rank

Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter‐rater reliability

Contact Info

Product

Resources

About