Can video improve grant review quality and lead to more reliable ranking?

Abstract

Multimedia video is rapidly becoming mainstream, and many studies indicate that it is a more effective communication medium than text. In this project we AIM to test if videos can be used, in place of text-based grant proposals, to improve communication and increase the reliability of grant ranking. We will test if video improves reviewer comprehension (AIM 1), if external reviewer grant scores are more consistent with video (AIM 2), and if mock Australian Research Council (ARC) panels award more consistent scores when grants are presented as videos (AIM 3). This will be the first study to evaluate the use of video in this application.The ARC reviewed over 3500 Discovery Project applications in 2015, awarding 635 Projects. Selecting the "best" projects is extremely challenging. This project will improve the selection process by facilitating the transition from text-based to video-based proposals. The impact could be profound: Improved video communication should streamline the grant preparation and review processes, enable more reliable ranking of applications, and more accurate identification of the "next big innovations". Keywordsresearch funding, grant writing, video, grant review, peer review, meta-research, research policy Aims and BackgroundIn the 21st century, multimedia video has revolutionised the way that we communicate throughout all aspects of life, and we reason that its incorporation into grant applications is an inevitable evolutionary next step. Annually, the Australian Research Council (ARC) and the National Health and Medical Research Council of Australia (NHMRC) review thousands of Project grant applications. Applicants' invest weeks into drafting text-based proposals designed to communicate the future of cutting edge science. The subsequent grant review processes are very challenging and time consuming, and prone to influence of reviewer variability. This results in a large "grey zone" where the success or failure of a grant can be significantly influenced by chance.Our previous research (focused on the NHMRC Project Grant review process) revealed that the top ~40% of awarded NHMRC Project Grant Applications were reliably identified as "fundable" when the variability in the current review processes was considered (Graves et al. 2011). However, the selection of the remaining ~60% of funded grants changed depending on the selected reviewers. Internationally, "luck of the draw" in the selection of peer reviewers has been repeatedly identified as major factor in funding and publication outcomes (Mayo et al. 2006, Osmond 1983, Smith 2006 We HYPOTHESISE that research proposals presented as recorded 10-minute PowerPoint videos, rather than traditional ARC text-based 10-page Project Descriptions, will lead to (1) a reduction in time required for applicants to prepare complex grant proposals, (2) more effective conveyance of grant concepts to reviewers, (3) more efficient reviews, enabling the engagement of more reviewers per application, and a reduction of the influence of reviewer bias, and ...