Construction and validation of a short multidisciplinary research performance questionnaire (SMRPQ)

Abstract

Research is often specialized and varies in its nature between disciplines, making it difficult to assess and compare the performance of individual researchers. Specific qualitative and quantitative indicators are usually complex and do not work equally well for different research fields. Therefore, the aim of the present study was to develop an economical questionnaire that is valid across disciplines. We constructed a Short Multidisciplinary Research Performance Questionnaire (SMRPQ), with which researchers can briefly report 11 quantitative and qualitative performance aspects from four areas (research quality, facilitation, transfer/exchange, and reputation) in relation to their peer reference groups (fellow researchers with the same status and discipline). To validate this questionnaire, 557 German researchers from Physics, History, and Psychology fields (53% male, 34% post-docs, and 19% full professors) completed it, and for the purpose of convergent and discriminant validation additionally made assessments regarding specific quantitative and qualitative indicators of research performance as well as affective, cognitive, and behavioural aspects of their research activities (perceptions of positive affect, help-seeking, and procrastination). The results attested reliable measurement, endorsed the postulated structure of the newly developed instrument, and confirmed its invariance across the three disciplines. The SMRPQ and the validation measure were strongly positively correlated, and both demonstrated similar associations with affect, cognition, and behaviour at work. Therefore, it can be considered a valid and economical approach for assessing research performance of individual researchers across different disciplines, especially within nomothetic research (e.g. regarding personal antecedents of successful research).

Dimensional comparisons in the formation of faculty members’ research and teaching self-concepts?

“…In accord with prior research (see Daumiller et al, 2019, for an overview), we found strong autoregressive effects for the examined constructs. This finding underpins the high stability of achievement and self-concept in research and teaching over time.…”

“…Achievements. We measured research achievement using three items from Daumiller et al (2019), in which the faculty members indicated whether they had published more publications, published higher-quality publications, or obtained more external funding during the last six months compared to faculty members in the same field and with the same academic status. The faculty members answered these items on a visual analog scale with sliders from 0 to 99 (e.g., "In the last six months, I have published more publications than ___% of the other researchers of my status in my community").…”

Teaching Quality in Higher Education

“…Second, we focused on a self-report measure to be used in nomothetic research. However, beyond such research, this approach is not appropriate for high stakes testing as it can be easily manipulated (see Daumiller, Siegel, et al, 2019), or for when individual dimensions of teaching quality are of interest. However, for teaching quality as a whole, we consider teacher self-reports as a valid indicator and find that, besides their face and predictive validity, they also converge well with student reports.…”

“…We used data from a larger research project in which student assessments of teaching quality were considered as consequences of faculty motivations (Daumiller, Siegel, et al, 2019). Students of the participating faculty members were asked to assess teaching quality across multiple sessions and courses, allowing for fine-grained insights.…”

On the issue of evaluating educational programs for additional professional education

“…Development and implementation of vocational educational programs. The central place in these issues is occupied by the tasks associated with developing and implementing educational programs for vocational education [1,2]. They are at the heart of any quality development, and the educational environment is no exception.…”