The Making and Evaluation of Digital Games Used for the Assessment of Attention: Systematic Review

Abstract

Background Serious games are now widely used in many contexts, including psychological research and clinical use. One area of growing interest is that of cognitive assessment, which seeks to measure different cognitive functions such as memory, attention, and perception. Measuring these functions at both the population and individual levels can inform research and indicate health issues. Attention is an important function to assess, as an accurate measure of attention can help diagnose many common disorders, such as attention-deficit/hyperactivity disorder and dementia. However, using games to assess attention poses unique problems, as games inherently manipulate attention through elements such as sound effects, graphics, and rewards, and research on adding game elements to assessments (ie, gamification) has shown mixed results. The process for developing cognitive tasks is robust, with high psychometric standards that must be met before these tasks are used for assessment. Although games offer more diverse approaches for assessment, there is no standard for how they should be developed or evaluated. Objective To better understand the field and provide guidance to interdisciplinary researchers, we aim to answer the question: How are digital games used for the cognitive assessment of attention made and measured? Methods We searched several databases for papers that described a digital game used to assess attention that could be deployed remotely without specialized hardware. We used Rayyan, a systematic review software, to screen the records before conducting a systematic review. Results The initial database search returned 49,365 papers. Our screening process resulted in a total of 74 papers that used a digital game to measure cognitive functions related to attention. Across the studies in our review, we found three approaches to making assessment games: gamifying cognitive tasks, creating custom games based on theories of cognition, and exploring potential assessment properties of commercial games. With regard to measuring the assessment properties of these games (eg, how accurately they assess attention), we found three approaches: comparison to a traditional cognitive task, comparison to a clinical diagnosis, and comparison to knowledge of cognition; however, most studies in our review did not evaluate the game’s properties (eg, if participants enjoyed the game). Conclusions Our review provides an overview of how games used for the assessment of attention are developed and evaluated. We further identified three barriers to advancing the field: reliance on assumptions, lack of evaluation, and lack of integration and standardization. We then recommend the best practices to address these barriers. Our review can act as a resource to help guide the field toward more standardized approaches and rigorous evaluation required for the widespread adoption of assessment games.

Effects of 1 Hz offline TMS on performance in the stop-signal game

“…Recently, several research teams have developed gamified stopsignal tasks (e.g., Schroeder et al, 2021), which do not change the underlying architecture of the SST but yield more enjoyable experiences for participants. Further, these gamified tasks provide a more engaging environment which creates a more captivating setting that may aid in collecting data from populations with a lower attention span such as children or groups of patients with concentration or attention deficits (Gallagher et al, 2023;Klock et al, 2020;Wiley et al, 2020Wiley et al, , 2021.…”

Design Preferences for a Serious Game–Based Cognitive Assessment of Older Adults in Prison: Thematic Analysis

“…Although there are well-evidenced benefits of using traditional cognitive assessment tools (proven psychometric validity, high sensitivity to neurological disease, etc) [4][5][6], they are often seen as boring and repetitive by users [7]. This can reduce task engagement [2] and affect the reliability of the performance data collected [1,8]. People also report test anxiety and self-stigma regarding low literacy or education levels when performing traditional cognitive tests [9][10][11][12].…”

“…Gamification stands as one of the most promising solutions to enhance the user experience of traditionally monotonous tasks [ 1 ]. The process of gamification includes applying game design elements such as challenges, scoring, graphics, and narratives to nongame environments (eg, psychometric testing) to increase engagement and motivation [ 2 ].…”

Digital Biomarkers for Well-being Through Exergame Interactions: Exploratory Study

“…On the health assessment front, the evaluation and validation of in-game metrics as assessment tools are performed either in comparison with a clinical diagnosis or a validated assessment test [ 25 ]. Cognitive measures in game-like interfaces can contribute to the early detection of neurological disease [ 26 ].…”

“…However, no further insights were provided regarding the well-being state of the participants. The study by Pirovano et al [ 25 ] developed an SG solution to support rehabilitation at home. Their solution combined fuzzy-based monitoring and in-game adaptation to capture the knowledge of the clinician and provide real-time feedback during exercise.…”