Validity Threats in Quantitative Data Collection With Games: A Narrative Survey

Abstract

Background. Games are increasingly used to collect scientific data. Some suggest that game features like high cognitive load may limit the inferences we can draw from such data, yet no systematic overview exists of potential validity threats of game-based methods. Aim. We present a narrative survey of documented and potential threats to validity in using games for quantitative data collection. Method. We combined an unsystematic bottom-up literature review with a systematic top-down application of standard validity threat typologies to games to arrive at a systematisation of game-characteristic validity threats. Results. We identify three game characteristics that potentially impact validity: Games are complex systems, impeding the predictable control and isolation of treatments. They are rich in unwanted variance and diversity. And their social framing can differ from and interact with the framing of research studies or non-game situations they are supposed to represent. The diversity of gamers and their differences to general populations bring further complications. Discussion and Conclusions. The wealth of potential validity threats in game-based research is met by a dearth of systematic methodological studies, leading us to outline several future research directions.

Item Quality Analysis for Measuring Mathematical Problem-Solving Skills

“…Tests can be categorized as a means of measuring an object by collecting interconnected information and fulfilling the characteristics of the object. In this study, the object in question was the cognitive level of students seen from the problem-solving skills, depending on the results of the stimulus that varies across types of stages or a game that can introduce the significant potential for confusion, namely arousal and cognitive load (Gundry & Deterding, 2019).…”

“…A test is said to have validity if the result corresponds to the criteria, in the sense of having a parallel between the test result and the requirements. In scientific research, validity is the primary construct and indicates the quality of the study (Gundry & Deterding, 2019).…”

Special Issue HCI Research Games – An Editorial

“…Geerts et al (2019) present three non-digital games they created for eliciting user needs and ideation purposes early in the design processes and discuss the structured approach they applied in designing and evaluating these research games. Finally, Gundry and Deterding (2019) identify the challenges posed by using games for data collection and how to deal with possible validity threats. of (board) game design elements in research, tools to facilitate non-technical end-users to engage in future thinking about complex technology, and participatory design for people with impairments.…”

Challenges and Opportunities of Field-based Data Collection with a Game. Analysis of the Development and use of a Game to Collect Data on People’s Emotional Experience in their Environment

“…Most game-based methods are used in research to collect data from large online populations (Gundry and Deterding, 2019). In field-based research, serious games are developed to simulate real life systems (Costanza et al, 2014).…”