Expert review method in game evaluations

Abstract

The expert review method is not yet widely adopted in game evaluations, although it has been used successfully in productivity software evaluations for years. In order to use the method effectively, there need to be playability heuristics that take into account the characteristics of the videogames. There are a few playability heuristic sets available, but they have several differences, and they have not been compared to discover their strengths and weaknesses in game evaluations. In this paper, we report on a first study comparing two playability heuristic sets when evaluating the playability of a videogame. The results indicate that the heuristics can assist the evaluators in evaluating both the user interface and the gameplay aspects of the game. However, playability heuristics need to be developed further before they can be utilized by the practitioners. Especially, the clarity and comprehensibility of the heuristics need to be improved, and the optimal number of heuristics is still open.

No need to read: Developing a pictorial single-item scale for measuring perceived usability

“…Several expert reviews were conducted at the beginning of the second phase to collect ideas for scale enhancements to improve its comprehensibility. This method is based on expert-based heuristic evaluation (Korhonen et al, 2009;Nielsen & Molich, 1990). The team involved in these reviews consisted of four experts in the domain of work psychology (half of them were also usability practitioners) and one expert in design.…”

Designing a Mobile Application for Working Memory Training through Understanding the Psychological and Physiological Characteristics of Older Adults

“…All users filled in the user experience questionnaire (UEQ) scale [41] after completing the task to record this product's strengths and weaknesses and further optimize and iterate on it. Experts were invited to complete a cognitive walk-through, simulate each operation step in the process of using the product, experience all testing tasks, review the product based on their professional knowledge and understanding during the experience [42], and make suggestions as well as modification comments.…”

Heuristic evaluation of an information literacy game

“…However, expert-based heuristic evaluations cannot provide attitudinal or behavioral data from actual users (Köffel & Haller, 2008). While experts are good at identifying general heuristic violations, users can point out more specific issues with the game design (Korhonen, Paavilainen, & Saarenpää, 2009). Evaluation with users is the benchmark of any playability evaluation (Desurvire et al, 2004).…”

“…The HEP has been validated in other studies. Korhonen, Paavilainen and Saarenpää (2009) used the HEP with eight experts from the game industry and academia. Results showed the HEP excelled in providing short and compact descriptions that were clear and understandable.…”