Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Automatically Defining Game Action Spaces for Exploration Using Program Analysis

“…Several prior works using automatic exploration invest human effort into modeling the game for the exploration agents. For example, work on reinforcement learning based exploration strategies manually defined the player's possible actions for the agents (Zheng et al 2019;Gordillo et al 2021;Liu et al 2022). Other approaches have manually constructed UML state machine models to generate test cases (Iftikhar et al 2015;Schaefer, Do, and Slator 2013).…”

“…We evaluated the impact of our approach for two popular exploration strategies. The first exploration strategy we used is random action selection, which has been commonly compared against as a baseline for automatic exploration of games (Liu et al 2022;Gordillo et al 2021;Zheng et al 2019;Zhan, Aytemiz, and Smith 2019) and mobile application testing (Choudhary, Gorla, and Orso 2015). In our evaluation, we used a random exploration policy that chooses uniformly at random from all valid actions in a given game state.…”

“…To provide a frame of reference for the performance of our approach, we additionally ran the same experiments with two other approaches discussed in the related work for providing the game actions to agents. First, we implemented the ideal case where we manually define each game's possible actions with a discrete action space, as done in prior work (Liu et al 2022;Zheng et al 2019), such that the action space covers all the functionality of the game as determined by a manual inspection of the game and its source code. Secondly, we define an action space which samples the common inputs on the platform's input device, as considered in previous work (Bellemare et al 2013;Zhan, Aytemiz, and Smith 2019).…”

“…In automatic exploration for games, an exploration agent automatically exercises different game mechanics with the aim of covering various game states or functionalities. This technique has been used for game testing, where different states are explored to reveal potential bugs (Zheng et al 2019;Gordillo et al 2021;Liu et al 2022), for amplifying the coverage of human tester data (Chang, Aytemiz, and Smith 2019), and for building a database of searchable moments (Zhan, Aytemiz, and Smith 2019).…”

“…Existing work has taken various approaches to determining the game actions. The most common approach is for developers to manually engineer an interface (Zheng et al 2019;Juliani et al 2020;Liu et al 2022) or model (Iftikhar et al 2015;Schaul 2013), however these approaches require additional developer effort. Another family of techniques guesses the possible game inputs based on the platform or input device (Bellemare et al 2013;Zhan, Aytemiz, and Smith 2019;Zhang et al 2018), however this approach can exhibit worse exploration performance due to many actions with invalid inputs having no effect.…”