Scenario-Based Approach for Blackbox Load Testing of Online Game Servers

Cho, Chang-Sik; Lee, Dong-Chun; Sohn, Kang-Min; Park, Chang-Joon; Kang, Ji-Hoon

doi:10.1109/cyberc.2010.54

Cited by 17 publications

(14 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As modern games continue growing both in size and complexity, game testing has become more challenging; consequently, even the most popular games on the market lack sufficient testing [1]- [3]. To improve efficiency and reduce cost, game testing has gradually evolved from manual and adhoc approaches to automated testing [5]- [23]. [5] proposed black box testing and scenario-based testing for online games.…”

Section: A Game Testingmentioning

confidence: 99%

Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Liu¹,

Cai²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) has attracted much attention in automated game testing. Early attempts rely on game internal information for game space exploration, thus requiring deep integration with games, which is inconvenient for practical applications. In this work, we propose using only screenshots/pixels as input for automated game testing and build a general game testing agent, Inspector, that can be easily applied to different games without deep integration with games. In addition to covering all game space for testing, our agent also tries to take human-like behaviors to interact with key objects in a game, since some bugs usually happen in playerobject interactions. Inspector is based on purely pixel inputs and comprises three key modules: game space explorer, key object detector, and human-like object investigator. Game space explorer aims to explore the whole game space by using a curiosity-based reward function with pixel inputs. Key object detector aims to detect key objects in a game, based on a small number of labeled screenshots. Human-like object investigator aims to mimic human behaviors for investigating key objects via imitation learning. We conduct experiments on two popular video games: Shooter Game and Action RPG Game 1 . Experiment results demonstrate the effectiveness of Inspector in exploring game space, detecting key objects, and investigating objects. Moreover, Inspector successfully discovers two potential bugs in those two games. The demo video of Inspector is available at https: //github.com/Inspector-GameTesting/Inspector-GameTesting.

show abstract

Section: A Game Testingmentioning

confidence: 99%

Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Liu¹,

Cai²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In the case of game software testing, these attributes include cross-platform operability, aesthetics, performance in terms of time and memory, consistency and functional correctness in a multi-user environment [3]. To validate these attributes, researchers proposed various methods to generate test sequences: record/replay [4], handcrafted scenarios [5], UML and state machines [6], Petri nets [7], and RL [8] [9]. However, when game environment changes, test sequences and scenarios such as [4] [5] become obsolete and a manual tester effort is required to create a new test sequence.…”

Section: Related Researchmentioning

confidence: 99%

“…Therefore, researchers proposed various methods to decrease the test effort. These methods are regression tests based on record/replay segments [4], scenario testing [5], UML-based sequence generation [6], agents harnessing Petri nets [7] and RL agents [8] [9] to automate this process. These studies, however, lack either human expertise, an automated oracle, an intelligent tester agent, or an overall game testing experiment.…”

Section: Introductionmentioning

confidence: 99%

Automated Video Game Testing Using Synthetic and Humanlike Agents

2021

View full text Add to dashboard Cite

In this paper, we present a new methodology that employs tester agents to automate video game testing. We introduce two types of agents -synthetic and human-like-and two distinct approaches to create them. Our agents are derived from Reinforcement Learning (RL) and Monte Carlo Tree Search (MCTS) agents, but focus on finding defects. The synthetic agent uses test goals generated from game scenarios, and these goals are further modified to examine the effects of unintended game transitions. The human-like agent uses test goals extracted by our proposed multiple greedy-policy inverse reinforcement learning (MGP-IRL) algorithm from tester trajectories. MGP-IRL captures multiple policies executed by human testers. These testers' aims are finding defects while interacting with the game to break it, which is considerably different from game playing. We present interaction states to model such interactions. We use our agents to produce test sequences, run the game with these sequences, and check the game for each run with an automated test oracle. We analyze the proposed method in two parts: we compare the success of human-like and synthetic agents in bug finding, and we evaluate the similarity between humanlike agents and human testers. We collected 427 trajectories from human testers using the General Video Game Artificial Intelligence (GVG-AI) framework and created three games with 12 levels that contain 45 bugs. Our experiments reveal that human-like and synthetic agents compete with human testers' bug finding performances. Moreover, we show that MGP-IRL increases the human-likeness of agents while improving the bug finding performance.

show abstract

“…To support developers with game testing, researchers have proposed several techniques. These include approaches to test the stability of game servers (e.g., by generating high packet loads) [19,20,30], model-based testing [29] using domain modeling for representing the game and UML state machines for behavioral modeling, as well as techniques specifically designed for testing board games [22,42]. When looking at recent techniques aimed at proposing more general testing frameworks, those exploiting Reinforcement Learning (RL) are on the rise.…”

Section: Introductionmentioning

confidence: 99%

Using Reinforcement Learning for Load Testing of Video Games

Tufano¹,

Scalabrino²,

Pascarella³

et al. 2022

Preprint

View full text Add to dashboard Cite

Different from what happens for most types of software systems, testing video games has largely remained a manual activity performed by human testers. This is mostly due to the continuous and intelligent user interaction video games require. Recently, reinforcement learning (RL) has been exploited to partially automate functional testing. RL enables training smart agents that can even achieve super-human performance in playing games, thus being suitable to explore them looking for bugs. We investigate the possibility of using RL for load testing video games. Indeed, the goal of game testing is not only to identify functional bugs, but also to examine the game's performance, such as its ability to avoid lags and keep a minimum number of frames per second (FPS) when highdemanding 3D scenes are shown on screen. We define a methodology employing RL to train an agent able to play the game as a human while also trying to identify areas of the game resulting in a drop of FPS. We demonstrate the feasibility of our approach on three games. Two of them are used as proof-of-concept, by injecting artificial performance bugs. The third one is an open-source 3D game that we load test using the trained agent showing its potential to identify areas of the game resulting in lower FPS.

show abstract

Scenario-Based Approach for Blackbox Load Testing of Online Game Servers

Cited by 17 publications

References 7 publications

Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Automated Video Game Testing Using Synthetic and Humanlike Agents

Using Reinforcement Learning for Load Testing of Video Games

Contact Info

Product

Resources

About