An Empirical Evaluation of Evaluation Metrics of Procedurally Generated Mario Levels

Abstract

There are several approaches in the literature for automatically generating Infinite Mario Bros levels. The evaluation of such approaches is often performed solely with computational metrics such as leniency and linearity. While these metrics are important for an initial exploratory evaluation of the content generated, it is not clear whether they are able to capture the player's perception of the content generated. In this paper we evaluate several of the commonly used computational metrics. Namely, we perform a systematic user study with procedural content generation systems and compare the insights gained from our user study with those gained from analyzing the computational metric values. The results of our experiment suggest that current computational metrics should not be used in lieu of user studies for evaluating content generated by computer programs.

The Right Variety: Improving Expressive Range Analysis with Metric Selection Methods

“…In future work we would like to explore whether metrics can be found which are highly performant across multiple game domains of similar genre. This could synergise well with papers discussed previously which explored the relationship between metrics and player perceptions [11,17,23]. Metric pairs that scored highly in multiple domains using our criteria, while also correlating with human perceptions of quality or diversity would have the potential to be robustly useful in novel PCG domains without the need for evaluation.…”

“…Summerville et al explored the power of common metrics when predicting player experience and found strong correlations [23]. Marino et al conducted a similar study and found much more limited correlation [17]. Herve & Salge explored the relationship between common metrics and expert evaluations of Minecraft maps [11].…”

An Examination of the Hidden Judging Criteria in the Generative Design in Minecraft Competition

“…Within the field of PCG, a truly qualitative evaluation of these artifacts is still challenging. Several experiments have already been conducted with the intent to critically examine some of the commonly used metrics and their relevance [10,8]. Although part of the tool set is pertinent in evaluation scenarios, it is also clear that we are missing player-driven evaluation methods.…”

Tile Embedding: A General Representation for Level Generation

“…For the VGLC representation, the output of the LSTM is connected to a dense layer with Softmax activation indicating the probability of a tile character. We perform an expressive range analysis of generated levels with the metrics: Linearity and Leniency (Summerville 2018;Smith et al 2010;Mariño, Reis, and Lelis 2015).…”