Assessing Learning from, with, and in Games Revisited: A Heuristic for Emerging Methods and Commercial Off-the-Shelf Games

Schrader, P. G.; McCreery, Michael P.; Carroll, Mark; Head, Danielle L.; Laferriere, Jeffrey R.

doi:10.1007/978-3-030-15569-8_2

Cited by 8 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The educational games literature has frameworks for conceptualizing and testing these processes that go well past the methods employed here (e.g. Schrader et al, 2019). We acknowledge that there are many researchers exploring relationships and meanings-making among players and teams rather than the causal effects we sought to measure (e.g.…”

Section: The Value Of Time Togethermentioning

confidence: 99%

See 1 more Smart Citation

The value of time together: a longitudinal investigation of mentor-protégé interactions in an online game

Williams

Choi

Sparks

et al. 2023

INTR

View full text Add to dashboard Cite

PurposeThe study aims to determine the outcomes of mentorship in an online game system, as well as the characteristics of good mentors.Design/methodology/approachA combination of anonymized survey measures and in-game behavioral measures were used to power longitudinal analysis over an 11-month period in which protégés and non-mentored new players could be compared for their performance, social connections and retention.FindingsSuccessful people were more likely to mentor others, and mentors increased protégés' skill. Protégés had significantly better retention, were more active and much more successful as players than non-protégés. Contrary to expectations, younger, less wealthy and educated people were more likely to be mentors and mentors did not transfer their longevity. Many of the qualities of the mentor remain largely irrelevant—what mattered most was the time spent together.Research limitations/implicationsThis is a study of an online game, which has unknown generalizability to other games and to offline settings.Practical implicationsThe results show that getting mentors to spend dedicated time with protégés matters more than their characteristics.Social implicationsGood mentorship does not require age or resources to provide real benefits.Originality/valueThis is the first study of mentorship to use survey and objective outcome measures together, over time, online.

show abstract

Section: The Value Of Time Togethermentioning

confidence: 99%

“…The educational games literature has frameworks for conceptualizing and testing these processes that go well past the methods employed here (e.g. Schrader et al. , 2019).…”

Section: Limitations Contributions and Looking Aheadmentioning

confidence: 99%

The value of time together: a longitudinal investigation of mentor-protégé interactions in an online game

Williams

Choi

Sparks

et al. 2023

INTR

View full text Add to dashboard Cite

show abstract

“…However, we chose to apply validation and reliability methods to COTS games, given the widespread use of COTS games for assessment in research (McCreery et al, 2019;Schrader et al, 2019;Shute et al, 2013). Commercial organizations are reducing the drawbacks of using GBAs by developing more cost-effective and accessible GBAs for organizational use (e.g., Arctic Shores, Revelian, Pymetrics).…”

Section: Limitationsmentioning

confidence: 99%

Virtual reality assessments (VRAs): Exploring the reliability and validity of evaluations in VR

Sanchez

Weiner²,

Zelderen

2022

Int J Selection Assessment

View full text Add to dashboard Cite

Virtual reality (VR) is a potential assessment format for constructs dependent on certain perceptual characteristics (e.g., realistic environment and immersive experience). The purpose of this series of studies was to explore methods of evaluating reliability and validity evidence for virtual reality assessments (VRAs) when compared with traditional assessments. We intended to provide the basis of a framework on how to evaluate VR assessments given that there are important fundamental differences to VR assessments compared with traditional assessment formats. Two commercial off-the-shelf (COTS) games (i.e., Project M and Richie's Plank Experience)were used in Studies 1 and 2, while a game-based assessment (GBA; Balloon Pop, designed for assessment) was used in Study 3. Studies 1 and 2 provided limited evidence for the reliability and validity of the VRAs. However, no meaningful constructs were measured by the VRA in Study 3. Findings demonstrate limited evidence for these VRAs as viable assessment options through the validity and reliability methods utilized in the present studies, which in turn emphasize the importance of aligning the assessment purpose to the unique advantages of a VR environment. K E Y W O R D Scommercial-off-the-shelf (COTS) games, game-based assessment (GBA), reliability, validity, virtual reality (VR) Practitioner points• Findings were mixed in correlating the VRA scores with similar assessments to the intended constructs being measured.• Details are provided on the design and scoring for the presented VRAs.• Although research using VRAs is still preliminary, there are promising methods through which we might design unique behavior based evaluation. | INTRODUCTIONThe measurement of trait differences using prehire assessments is widely practiced and useful to help differentiate job candidates based on constructs that are related to job performance, such as personality traits like conscientiousness (r = .31) and aptitudes like cognitive ability (r = .51;Hough et al., 2001). The self-report method has demonstrated accurate measurement for certain constructs in various contexts (Chan, 2009

show abstract

“…Broadly speaking, that variability manifests through individual differences (e.g., cognitive ability) intersecting with contextualized experiences to produce an outcome (Goh et al, 2019;Jonassen & Grabowski, 2012). For this reason, researchers have begun to surmise that the complexity of video games, coupled with content embedded in narrative, may provide a contextualized experience through which individual differences shape learning (Schrader et al, 2019).…”

Section: Objectives and Purpose Of The Studymentioning

confidence: 99%

“…Briefly, complex systems are multifaceted (multiplicity of components), dynamic (interaction of multiple components), and emergent (evolving over time) (Hilpert & Marchand, 2018). Video games are complex systems consisting of a variety of elements (e.g., dimensional space, GUI, world objects) that afford interactions with each other and the player over time (Schrader et al, 2019). The player must possess a basic level of skill and understanding within this complex system in order to attend to and accurately interpret the feedback provided by interactions with the rules, goals, and objectives (McGonigal, 2011).…”

Section: Theoretical Frameworkmentioning

confidence: 99%

Time to Acclimate: Examining the Influence of Cognitive Ability on Situating to a Video Game

McCreery¹,

Head²,

Fiorentini³

et al. 2020

Proceedings of the 17th International Conference on Cognition and Exploratory Learning in the Digital Age (CELDA 2020)

View full text Add to dashboard Cite

The current study was designed to examine the relationship between cognitive ability and player experience in shaping how players think within, and acclimate to, video games as complex systems. Specifically, researchers examined the relationship between player's cognitive ability and gameplay outcomes within a video game (i.e., The Deed). Outcomes were evaluated and contrasted after two discrete playthroughs, each lasting approximately 30 minutes. Logistic regression indicated that cognitive ability predicted individual outcomes for both the first and second playthrough, but did not predict growth between playthroughs (i.e., outcome change). Findings are then discussed in terms of an acclimation stage.

show abstract

Assessing Learning from, with, and in Games Revisited: A Heuristic for Emerging Methods and Commercial Off-the-Shelf Games

Cited by 8 publications

References 38 publications

The value of time together: a longitudinal investigation of mentor-protégé interactions in an online game

The value of time together: a longitudinal investigation of mentor-protégé interactions in an online game

Virtual reality assessments (VRAs): Exploring the reliability and validity of evaluations in VR

Time to Acclimate: Examining the Influence of Cognitive Ability on Situating to a Video Game

Contact Info

Product

Resources

About