Development and validation of a questionnaire to measure research impact

Abstract

Although questionnaires are widely used in research impact assessment, their metric properties are not well known. Our aim is to test the internal consistency and content validity of an instrument designed to measure the perceived impacts of a wide range of research projects. To do so, we designed a questionnaire to be completed by principal investigators in a variety of disciplines (arts and humanities, social sciences, health sciences, and information and communication technologies). The impacts perceived and their associated characteristics were also assessed. This easy-to-use questionnaire demonstrated good internal consistency and acceptable content validity. However, its metric properties were more powerful in areas such as knowledge production, capacity building and informing policy and practice, in which the researchers had a degree of control and influence. In general, the research projects represented an stimulus for the production of knowledge and the development of research skills. Behavioural aspects such as engagement with potential users or mission-oriented projects (targeted to practical applications) were associated with higher social benefits. Considering the difficulties in assessing a wide array of research topics, and potential differences in the understanding of the concept of ‘research impact’, an analysis of the context can help to focus on research needs. Analyzing the metric properties of questionnaires can open up new possibilities for validating instruments used to measure research impact. Further to the methodological utility of the current exercise, we see a practical applicability to specific contexts where multiple discipline research impact is requires.

A cross‐sectional multicenter survey on the future of dental education in the era of COVID‐19: Alternatives and implications

“…Nevertheless, our questionnaire was designed to be simple, concise, and precise, taking only about 10 minutes to complete. Previous surveys were referred to for general guidance 21,22 . To encourage the invited faculty members to join the survey, clear statements were used in the email invitations, and the survey link to complete the survey was anonymous and voluntary with the option to leave at any time.…”

Using chi-squared automatic interaction detection modelling to identify student opinion profiles regarding same-sex couples as a family structure

“…In all cases, correlation coefficients of r > 0.35 were achieved. According to some scientists ( Cook and Beckman, 2006 ; Heale and Twycross, 2015 ; Campos et al., 2017 ; Solans-Domènech et al., 2019 ), this is sufficient empirical evidence to be able to confirm that each item individually contributes to the scale (univocity). Finally, exploratory factor analysis was performed with a double objective.…”

Evaluating Research Centers in Minority Institutions: Framework, Metrics, Best Practices, and Challenges

“…Given the complexity of what (and how) research efforts are translated to the community, funders should acknowledge the approaches that allow evaluation of earlier phases of research and research centers' program activities. A focus on a variety of process measures is critical for demonstrating the broad range of impact of research centers [2,6]. The focus on processes, along with outcomes and impacts when available, provides insight on how an academic research center may influence research activities, starting at the level of investigators directly involved in the center [7].…”

“…The National Institutes of Health (NIH) has various types of grants to support research-related programs and funds over 1200 research centers at $41.7 billion USD annually [1]. The NIH expects these research centers to demonstrate how they ultimately contribute to impactful science [2,3]. Ideally, research centers demonstrate both an 'academic impact' of fostering research, with a 'wider and societal impact' that includes benefits outside of academia, e.g., for communities [2].…”

“…The NIH expects these research centers to demonstrate how they ultimately contribute to impactful science [2,3]. Ideally, research centers demonstrate both an 'academic impact' of fostering research, with a 'wider and societal impact' that includes benefits outside of academia, e.g., for communities [2]. NIH research centers are funded with collective goals to decrease morbidity and mortality [4].…”