Virtual reality assessments (VRAs): Exploring the reliability and validity of evaluations in VR

Abstract

Virtual reality (VR) is a potential assessment format for constructs dependent on certain perceptual characteristics (e.g., realistic environment and immersive experience). The purpose of this series of studies was to explore methods of evaluating reliability and validity evidence for virtual reality assessments (VRAs) when compared with traditional assessments. We intended to provide the basis of a framework on how to evaluate VR assessments given that there are important fundamental differences to VR assessments compared with traditional assessment formats. Two commercial off-the-shelf (COTS) games (i.e., Project M and Richie's Plank Experience)were used in Studies 1 and 2, while a game-based assessment (GBA; Balloon Pop, designed for assessment) was used in Study 3. Studies 1 and 2 provided limited evidence for the reliability and validity of the VRAs. However, no meaningful constructs were measured by the VRA in Study 3. Findings demonstrate limited evidence for these VRAs as viable assessment options through the validity and reliability methods utilized in the present studies, which in turn emphasize the importance of aligning the assessment purpose to the unique advantages of a VR environment. K E Y W O R D Scommercial-off-the-shelf (COTS) games, game-based assessment (GBA), reliability, validity, virtual reality (VR) Practitioner points• Findings were mixed in correlating the VRA scores with similar assessments to the intended constructs being measured.• Details are provided on the design and scoring for the presented VRAs.• Although research using VRAs is still preliminary, there are promising methods through which we might design unique behavior based evaluation. | INTRODUCTIONThe measurement of trait differences using prehire assessments is widely practiced and useful to help differentiate job candidates based on constructs that are related to job performance, such as personality traits like conscientiousness (r = .31) and aptitudes like cognitive ability (r = .51;Hough et al., 2001). The self-report method has demonstrated accurate measurement for certain constructs in various contexts (Chan, 2009

Game-related assessments for personnel selection: A systematic review

“…Their results show that, as with the original version, the associations with the Big Five measured with a conventional test are modest, and there are no significant differences between the two versions of the game. We also include herein the work of Sanchez et al (2022) , who use playful games to evaluate sensation-seeking, height aversion, and risk-taking, only finding a relationship between the game scores when evaluating the first two constructs and openness to experience. The latest study on construct validity, specifically on personality, offers unfavorable results: Wu et al (2022) develop two game-based assessments to evaluate facets of conscientiousness but which really assess cognitive ability and the remaining Big Five.…”

“…The game scores show a moderate association with a situational judgment test. Sanchez et al (2022) examine whether a playful game of virtual reality can be used to evaluate emotional intelligence, finding a moderate relationship with a conventional test. In fact, the associations with measures of personality turned out to be higher than the associations with emotional intelligence.…”

“…(1) gamified assessment (e.g., Hommel et al, 2022); (2) gamefully designed assessment (e.g., Georgiou, 2021); (3) game-based assessments (e.g., Wiernik et al, 2022); and (4) playful games used for assessment purposes (e.g., Sanchez et al, 2022).…”

“…As shown in Figure 1 , the classification begins on a continuum: at one end are the traditional assessments (e.g., tests, simulations), and at the other end, the playful games created for fun. Thus, we distinguish at least four types of GRA: (1) gamified assessment (e.g., Hommel et al, 2022 ); (2) gamefully designed assessment (e.g., Georgiou, 2021 ); (3) game-based assessments (e.g., Wiernik et al, 2022 ); and (4) playful games used for assessment purposes (e.g., Sanchez et al, 2022 ).…”

Assessing Electroencephalography as a Stress Indicator: A VR High-Altitude Scenario Monitored through EEG and ECG

“…Over the last decade, multiple studies have used VR environments to evoke emotional reactions [ 22 , 37 ]. Eye tracking [ 38 ], questionnaires [ 39 ], and respiration signals and ECG [ 40 ] are the most common measures used to assess stress in VR-related studies, while EEG has seldom been used [ 41 ]. One key explanation for this is that the combination of an EEG and a VR headset is not easily achievable due to device placement overlapping restrictions.…”