Investigating the construct validity of an electronic in-basket exercise using bias-corrected bootstrapping and Monte Carlo re-sampling techniques

Abstract

Orientation: Technology-based simulation exercises are popular assessment measures for the selection and development of human resources.Research purpose: The primary goal of this study was to investigate the construct validity of an electronic in-basket exercise using computer-based simulation technology. The secondary goal of the study was to investigate how re-sampling techniques can be used to recover model parameters using small samples.Motivation for the study: Although computer-based simulations are becoming more popular in the applied context, relatively little is known about the construct validity of these measures.Research approach/design and method: A quantitative ex post facto correlational design was used in the current study with a convenience sample (N = 89). The internal structure of the simulation exercise was assessed using a confirmatory factor analytical approach. In addition, bias-corrected bootstrapping and Monte Carlo simulation strategies were used to assess the confidence intervals around model parameters.Main findings: Support was not found for the entire model, but only for one of the dimensions, namely, the Interaction dimension. Multicollinearity was found between most of the dimensions that were problematic for factor analyses.Practical/managerial implications: This study holds important implications for assessment practitioners who hope to develop unproctored simulation exercises.Contribution/value-add: This study aims to contribute to the existing debate regarding the validity and utility of assessment centres (ACs), as well as to the literature concerning the use of technology-driven ACs. In addition, the study aims to make a methodological contribution by demonstrating how re-sampling techniques can be used in small AC samples.

Objective Structured Assessment of Debriefing (OSAD) in simulation-based medical education: Translation and validation of the German version

“…Based on recommendations [ 29 ], to define an adequate fit, the standardised root mean square residual and root mean square error of approximation should be <0.05, and the normed fit index, comparative fit index and Tucker–Lewis index should be >0.95. We performed bootstrapping [ 30 ] and applied Monte-Carlo re-sampling techniques to test the appropriateness of the model parameters, standard errors, confidence intervals and fit indices [ 30 , 31 ], because the sample size was only 57 debriefings. Finally, internal consistency was evaluated using Cronbach’s alpha (α), and composite reliability using Omega (ω) [ 32 ].…”