Investigating the Performance of Various Deep Neural Networks-based Approaches Designed to Identify Game Events in Gameplay Footage

Abstract

Video games, in addition to representing an extremely relevant field of entertainment and market, have been widely used as a case study in artificial intelligence for representing a problem with a high degree of complexity. In such studies, the investigation of approaches that endow player agents with the ability to retrieve relevant information from game scenes stands out, since such information can be very useful to improve their learning ability. This work proposes and analyses new deep learning-based models to identify game events occurring in Super Mario Bros gameplay footage. The architecture of each model is composed of a feature extractor convolutional neural network (CNN) and a classifier neural network (NN). The extracting CNN aims to produce a feature-based representation for game scenes and submit it to the classifier, so that the latter can identify the game event present in each scene. The models differ from each other according to the following elements: the type of the CNN; the type of the NN classifier; and the type of the game scene representation at the CNN input, being either single frames, or chunks, which are n-sequential frames (in this paper 6 frames were used per chunk) grouped into a single input. The main contribution of this article is to demonstrate the greater performance reached by the models which combines the chunk representation for the game scenes with the resources of the classifier recurrent neural networks (RNN).

Proposing an Automatic System for Generating Super Mario Bros Datasets

“…For this purpose, the present paper proposes the automation of data generation in the SMB Framework through video frames and game logs, which register what the players are doing, as well as what is happening in the environment. These logs correspond to game information that efficiently represents the players' experiences, as already validated in recent works [Luo et al 2018, Green et al 2020, Faria et al 2022]. The authors here expect that this will speed up the development and validation of future PCG methods, for example, to create personalized levels based on players' mechanics [Green et al 2020].…”

“…. These logs provide information about the agent's interaction with the game and were defined mainly based on the work [Faria et al 2022] in which the main objective was to identify game events in gameplay footage through deep learning methods. The authors of the present paper believe that this kind of dataset can be very useful to assist in the evolution of AI research carried out on the SMB game.…”