Background. Games are increasingly used to collect scientific data. Some suggest that game features like high cognitive load may limit the inferences we can draw from such data, yet no systematic overview exists of potential validity threats of game-based methods. Aim. We present a narrative survey of documented and potential threats to validity in using games for quantitative data collection. Method. We combined an unsystematic bottom-up literature review with a systematic top-down application of standard validity threat typologies to games to arrive at a systematisation of game-characteristic validity threats. Results. We identify three game characteristics that potentially impact validity: Games are complex systems, impeding the predictable control and isolation of treatments. They are rich in unwanted variance and diversity. And their social framing can differ from and interact with the framing of research studies or non-game situations they are supposed to represent. The diversity of gamers and their differences to general populations bring further complications. Discussion and Conclusions. The wealth of potential validity threats in game-based research is met by a dearth of systematic methodological studies, leading us to outline several future research directions.

Applied games are increasingly used to collect human subject data such as people's performance or attitudes. Games aford a motive for data provision that poses a validity threat at the same time: as players enjoy winning the game, they are motivated to provide dishonest data if this holds a strategic in-game advantage. Current work on data collection game design doesn't address this issue. We therefore propose a theoretical model of why people provide certain data in games, the Rational Game User Model. We derive a design approach for human subject data collection games that we call Intrinsic Elicitation: data collection should be integrated into the game's mechanics such that honest responding is the necessary, strategically optimal, and least efortful way to pursue the game's goal. We illustrate the value of our approach with a sample analysis of the data collection game Urbanology.

Idle games-games where waiting for extended periods is an important dynamic-are increasing in popularity. The game Neko Atsume, a mobile game about collecting cats, is an extreme example of this genre where progress can only be achieved when the game is switched off (so-called "progress while gone"). Do such waiting games engage players? To answer this, we conducted a large survey (N=1972) to understand what players are doing when they play Neko Atsume. Players are highly "engaged" in Neko Atsume as they interact with and around the game in four distinct ways: Time spent playing, Direct sociability; Social media sociability and Checking frequency. However, these characteristics of engagement in Neko Atsume do not fit well with existing models of engagement. We propose that, in future studies, game engagement in idle games could be considered as a habit which players acquire and maintain.

Games have become a popular way of collecting human subject data, based on the premise that they are more engaging than surveys or experiments, but generate equally valid data. However, this premise has not been empirically tested. In response, we designed a game for eliciting linguistic data following Intrinsic Elicitation – a design approach aiming to minimise validity threats in data collection games – and compared it to an equivalent linguistics experiment as control. In a preregistered study and replication (n=96 and n=136), using two different ways of operationalising accuracy, the game generated substantially more enjoyment (d=.70, .73) and substantially less accurate data (d=-.68, -.40) – though still more accurate than random responding. We conclude that for certain data types data collection games may present a serious trade-of between participant enjoyment and data quality, identify possible causes of lower data quality for future research, reflect on our design approach, and urge games HCI researchers to use careful controls where appropriate.