Cognitive ability in virtual reality: Validity evidence for VR game‐based assessments

Abstract

Virtual reality (VR) technology may provide unique benefits for assessing individual characteristics, in contrast with traditional assessments. This study examined validity evidence for the use of VR game (VRG) scores to indicate specific cognitive abilities. Participants completed three VRGs, three computer‐based assessments (CBAs), and additional scales. To evaluate convergent validity, VRG scores were compared with CBA scores. For divergent validity, VRG scores were compared with five factors of personality based on the Five Factor Model. For criterion‐related validity, VRG scores were compared with academic achievement (i.e., grade point average). Findings generally provided support that the VRG scores are related to certain spatial reasoning subdimensions of cognitive ability. However, further research is needed to examine the qualities of VRGs as assessment tools.

Individual differences at play: An investigation into measuring Big Five personality facets with game‐based assessments

“…This finding indicates that important research still needs to be conducted on designing GBAs with lower cognitive ability requirements or correlates, while preserving enough complexity in gameplay to allow for manifestations of trait-relevant behaviors. Conversely, GBAs designed for measuring cognitive ability may be predictive of personality traits, such as facets of openness to experience (Weiner & Sanchez, 2020) versa. Another noteworthy finding is that both GBAs differed on their levels of predictions for each type of cognitive ability (Click Town was more predictive of matrix reasoning; and Word Find was more predictive of verbal ability).…”

“…Well known in the traditional assessment literature is how cognitive ability assessments in personnel selection can result in adverse impact, in particular for ethnic minorities (Roth et al, 2001). Generally, it is less clear how much adverse impact is found in cognitively loaded or cognitive ability-oriented GBAs (Bhatia & Ryan, 2018;Weiner & Sanchez, 2020). In addition to the adverse impact associated with cognitive ability measures, GBAs may also magnify adverse impact for females in terms of video game experience, considering that males have been found to not only play video games more often on but also for longer periods of time than females (Bhatia & Ryan, 2018;Gough, 2020aGough, , 2020b.…”

“…GBA researchers have continued to investigate which psychological constructs can be tapped intentionally or indirectly by GBAs. For example, previous research has supported certain in-game behaviors as either reliable measures or strong correlates of various cognitive abilities, including abstract reasoning (fluid intelligence), knowledge (crystallized intelligence), inductive reasoning, visual perception, memory, learning, and cognitive speed (Quiroga & Colom, 2019;Quiroga et al, 2015;Weidner & Short, 2019;Weiner & Sanchez, 2020). Sometimes, GBAs may intentionally measure cognitive ability constructs, to the point that they may even look like a traditional cognitive ability test; whereas in many other cases, the GBA might appear more game-like, and the GBA's cognitive components are discovered through their correlations with traditional cognitive ability measures.…”

“…Likewise, certain GBAs also appear to measure personality, given that in-game behaviors look like and correlate with traditional measures of personality traits, such as conscientiousness, extraversion, emotional stability, and honesty-humility (see Weidner & Short, 2019). Even in the case of some GBAs expressly designed to measure cognitive ability, researchers have found relations between GBA predictors and facets of openness to experience, such as imagination (Weiner & Sanchez, 2020).…”

Defining motivation in video game‐based training: Exploring the differences between measures of motivation

“…Other recent research has explored new technologybased solutions to training and assessment such as automated feedback and VR interventions (e.g. Langer et al, 2019;Weiner & Sanchez, 2020). This recommendation is provided while acknowledging that good instructional design will likely the impact of media differences in training interventions 1994).…”