Peering at peer review revealed high degree of chance associated with funding of grant applications

Mayo, Nancy E.; Brophy, James M.; Goldberg, Mark S.; Klein, Marina B.; Miller, Sydney B.; Platt, Robert W.; Ritchie, Judith A.

doi:10.1016/j.jclinepi.2005.12.007

Cited by 63 publications

(65 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the enormous expenditure, there is surprisingly little research conducted on how this process could be optimised. The role of chance in funding outcomes is well known to researcher, and increasingly well described in the literature (Graves et al 2011, Mayo et al 2006, Osmond 1983, Herbert et al 2013. Our studies indicated that ~60% of Table 1.…”

Section: Significance Innovation National Benefit and Feasibility Smentioning

confidence: 53%

“…Enhanced communication will ultimately ease pressure on reviewers/panel members, and allow them to focus on evaluating and ranking the relative the scientific merit of each application. Chance has been repeatedly identified as a problem in funding and publication peer review (Mayo et al 2006, Osmond 1983. Richard Smith, editor of the BMJ from 1992 to 2004, called peer review "largely a lottery" (Smith 2006).…”

Section: Resultsmentioning

confidence: 99%

“…However, the selection of the remaining ~60% of funded grants changed depending on the selected reviewers. Internationally, "luck of the draw" in the selection of peer reviewers has been repeatedly identified as major factor in funding and publication outcomes (Mayo et al 2006, Osmond 1983, Smith 2006 We HYPOTHESISE that research proposals presented as recorded 10-minute PowerPoint videos, rather than traditional ARC text-based 10-page Project Descriptions, will lead to (1) a reduction in time required for applicants to prepare complex grant proposals, (2) more effective conveyance of grant concepts to reviewers, (3) more efficient reviews, enabling the engagement of more reviewers per application, and a reduction of the influence of reviewer bias, and (4) greater confidence in the system's capacity to appropriately allocate research funding. (Smith 2006).…”

Section: Aims and Backgroundmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Can video improve grant review quality and lead to more reliable ranking?

Doran¹,

Barnett²,

Leach³

et al. 2017

RIO

View full text Add to dashboard Cite

Multimedia video is rapidly becoming mainstream, and many studies indicate that it is a more effective communication medium than text. In this project we AIM to test if videos can be used, in place of text-based grant proposals, to improve communication and increase the reliability of grant ranking. We will test if video improves reviewer comprehension (AIM 1), if external reviewer grant scores are more consistent with video (AIM 2), and if mock Australian Research Council (ARC) panels award more consistent scores when grants are presented as videos (AIM 3). This will be the first study to evaluate the use of video in this application.The ARC reviewed over 3500 Discovery Project applications in 2015, awarding 635 Projects. Selecting the "best" projects is extremely challenging. This project will improve the selection process by facilitating the transition from text-based to video-based proposals. The impact could be profound: Improved video communication should streamline the grant preparation and review processes, enable more reliable ranking of applications, and more accurate identification of the "next big innovations". Keywordsresearch funding, grant writing, video, grant review, peer review, meta-research, research policy Aims and BackgroundIn the 21st century, multimedia video has revolutionised the way that we communicate throughout all aspects of life, and we reason that its incorporation into grant applications is an inevitable evolutionary next step. Annually, the Australian Research Council (ARC) and the National Health and Medical Research Council of Australia (NHMRC) review thousands of Project grant applications. Applicants' invest weeks into drafting text-based proposals designed to communicate the future of cutting edge science. The subsequent grant review processes are very challenging and time consuming, and prone to influence of reviewer variability. This results in a large "grey zone" where the success or failure of a grant can be significantly influenced by chance.Our previous research (focused on the NHMRC Project Grant review process) revealed that the top ~40% of awarded NHMRC Project Grant Applications were reliably identified as "fundable" when the variability in the current review processes was considered (Graves et al. 2011). However, the selection of the remaining ~60% of funded grants changed depending on the selected reviewers. Internationally, "luck of the draw" in the selection of peer reviewers has been repeatedly identified as major factor in funding and publication outcomes (Mayo et al. 2006, Osmond 1983, Smith 2006 We HYPOTHESISE that research proposals presented as recorded 10-minute PowerPoint videos, rather than traditional ARC text-based 10-page Project Descriptions, will lead to (1) a reduction in time required for applicants to prepare complex grant proposals, (2) more effective conveyance of grant concepts to reviewers, (3) more efficient reviews, enabling the engagement of more reviewers per application, and a reduction of the influence of reviewer bias, and ...

show abstract

Section: Significance Innovation National Benefit and Feasibility Smentioning

confidence: 53%

Section: Resultsmentioning

confidence: 99%

Section: Aims and Backgroundmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Can video improve grant review quality and lead to more reliable ranking?

Doran¹,

Barnett²,

Leach³

et al. 2017

RIO

View full text Add to dashboard Cite

show abstract

“…Even if they are not primary or secondary reviewers, it is felt that all members of a grant review panel should have the opportunity to read the grant application and participate in the discussion and scoring. 10,11 This removes the impact of extreme reviews and produces more consistency. At the same time, analysis has shown that if all members of the panel are equally responsible for reviewing an application, reviews can be more superficial than thorough and critical.…”

Section: The Review Panelmentioning

confidence: 99%

Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association

Liaw¹,

Freedman²,

Becker³

et al. 2017

Circulation Research

View full text Add to dashboard Cite

The biomedical research enterprise depends on the fair and objective peer review of research grants, leading to the distribution of resources through efficient and robust competitive methods. In the United States, federal funding agencies and foundations collectively distribute billions of dollars annually to support biomedical research. For the American Heart Association, a Peer Review Subcommittee is charged with establishing the highest standards for peer review. This scientific statement reviews the current literature on peer review practices, describes the current American Heart Association peer review process and those of other agencies, analyzes the strengths and weaknesses of American Heart Association peer review practices, and recommends best practices for the future

show abstract

Moving beyond the boilerplate: Reflections on equity‐centered reviewing for granting organizations

Marshall

Salter

2022

Science Education

View full text Add to dashboard Cite

This essay centers on the voices of leading scholars in science and STEM education on how equity can and should be centered in reviewing proposals for granting organizations. As the decisions made as a result of the reviewing process significantly impact the future directions of the field, we recognize the urgency in considering how equity is considered in this process. Through their experiences, four researchers offer the science and STEM community a call to action. The scholars interviewed highlighted that equitable reviewing and funding research and professional development will require changes within the science education and STEM funding ecosystem. Three overarching themes include (1) changing the ideologies and culture of science and STEM education research funding will require centering the needs of the communities being served; (2) institutions and granting organizations should adopt equity‐focused and holistic rubrics and models; and 3) we each have an individual responsibility to employ equity during the review process. Thus, this essay has the potential to both inspire and provide explicit examples of how we can all center equity as we strive to transform the future of science and STEM education.

show abstract

Peering at peer review revealed high degree of chance associated with funding of grant applications

Cited by 63 publications

References 33 publications

Can video improve grant review quality and lead to more reliable ranking?

Can video improve grant review quality and lead to more reliable ranking?

Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association

Moving beyond the boilerplate: Reflections on equity‐centered reviewing for granting organizations

Contact Info

Product

Resources

About