The development of a four-tier test to evaluate research integrity training

Abstract

Although higher education institutions across Europe and beyond are paying more and more attention to research integrity training, there are few studies and little evidence on what works and what does not work in such training. One way to overcome this challenge is to evaluate such training with standardised instruments. Experts/trainers have used qualitative approaches to evaluate their research integrity training's successes, but it is difficult to compare their results with others. Sometimes they conduct standardised tests drawn from ethics education or other related fields, but these tests do not assess research integrity's core themes. At present, there is a lack of standardised instruments designed to specifically evaluate success in research integrity training. This article presents a pre-validated instrument for this purpose. Named the P2I questionnaire, it is designed as a four-tier test and based on the European Code of Conduct for Research Integrity; in it, testees choose a (scientific) practice to address an issue, justify their decision, and describe how confident they are with their decisions. The development of the P2I questionnaire is outlined in three steps. After describing the research integrity (alternatively, the responsible conduct of research) training successes, the article notes scientific and non-scientific patterns and then concludes with a pre-validated and revised version of the P2I questionnaire. This questionnaire is intended as a first step in a discourse on standardised research integrity measurements and is one step towards an evidence-based improvement of research integrity training.

Fostering a research integrity culture: Actionable advice for institutions

“…6. Although ethics and integrity can be differentiated (see e.g., Marusic et al 2016;Valkenburg et al 2021;Zollitsch et al 2021) and may require separate training, there is substantial overlap and in particular in the early stages of integrity education and training development, the terms have been used interchangeably in the literature. We use 'RCR training' as an umbrella term.…”

“…Because institutions are, in most cases, organizing trainings themselves they are optimally positioned to design studies and collect empirical data about the effectiveness of trainings and about how they are evaluated by trainees. Nonetheless, although pre-post designs with control groups provide the most conclusive evidence for program effectiveness, a multitude of designs and instruments is used and often only one or two outcome criteria are used (e.g., Antes et al 2009;Marusic et al 2016;McIntosh et al 2018;Mumford et al 2015;Steele et al 2016;Zollitsch et al 2021). As a result, the available evidence is often unreliable, with a high risk of bias (Marusic et al 2016;Zollitsch et al 2021).…”

“…Nonetheless, although pre-post designs with control groups provide the most conclusive evidence for program effectiveness, a multitude of designs and instruments is used and often only one or two outcome criteria are used (e.g., Antes et al 2009;Marusic et al 2016;McIntosh et al 2018;Mumford et al 2015;Steele et al 2016;Zollitsch et al 2021). As a result, the available evidence is often unreliable, with a high risk of bias (Marusic et al 2016;Zollitsch et al 2021). Therefore, Mumford and colleagues (2015) proposed a 'multilevel approach' in which outcome measures are tailored to the training objectives, i.e.…”

“…Again, some researchers have suggested that a database of evaluation studies would allow identifying the most effective RCR training interventions and their most effective target population and delivery methods through meta-analysis (Marusic et al 2016;Mumford et al 2015;Phillips et al 2018;Steele et al 2016;Zollitsch et al 2021). Todd and colleagues (2017) add that multiple process-based variables have not yet been used and are hence not included in meta-analyses -so we do not even know whether the most indicative measures are used.…”