Scoping Analytical Usability Evaluation Methods: A Case Study

Abstract

Analytical usability evaluation methods (UEMs) can complement empirical evaluation of systems: for example, they can often be used earlier in design, and can provide accounts of why users might experience difficulties, as well as what those difficulties are. However, their properties and value are only partially understood. One way to improve our understanding is by detailed comparisons using a single interface or system as a target for evaluation, but we need to look deeper than simple problem counts: we need to consider what kinds of accounts each UEM offers, and why. Here, we report on a detailed comparison of eight analytical UEMs. These eight methods were applied to a robotic arm interface, and the findings were systematically compared against video data of the arm in use. The usability issues that were identified could be grouped into five categories: system design; user misconceptions; conceptual fit between user and system; physical issues; and contextual ones. Other possible categories such as user experience did not emerge in this particular study. With the exception of Heuristic Evaluation, which supported a range of insights, each analytical method was found to focus attention on just one or two categories of issues. Two of the three 'home grown' methods (EMU and CASSM) were found to occupy particular niches in the space while the third (PUM) did not. This approach has identified commonalities and contrasts between methods and provided accounts of why a particular method yielded the insights it did. Rather than considering measures such as problem count or thoroughness, this approach has yielded insights into the scope of each method.

Identifying Phenotypes and Genotypes: A Case Study Evaluating an In-Car Navigation System

“…[10] is a methodology developed to address specifically the evaluation of usability of multi-modal systems. The scope of the methodology extends to issues related to user knowledge and use in context, with a special focus on the issues concerned with the physical relationship between the user and the device [1]. Multimodal interfaces can enhance the user's understanding, as they provide more communication channels between the user and the system.…”

“…Various case studies are reported in the literature with regard to the evaluation of usability methods ( [18], [1], [12], [4], [6]). Comparisons between methods have been carried out in terms of problem count, scope, validity, evaluator effect, etc.…”

Exploring organisational competences in Human Factors and UX project work: managing careers, project tactics and organisational strategy

“…Methods must be chosen to provide leverage on the issues a client faces (Blandford et al 2008) within the constraints of the project. For example, workload methods should be chosen for workload problems, and analytic rather than empirical methods may be selected if the project is fast and cheap.…”

EMU in the Car: Evaluating Multimodal Usability of a Satellite Navigation System

“…In particular, clashes between modalities, integration of information, and possible lost information can be identified through an EMU analysis. Earlier studies have shown that EMU is learnable [10] and that it occupies a particular niche within the space of analytical evaluation methods [1]. However, the initial study of EMU in use highlighted the fact that the process of conducting an analysis was unduly laborious.…”

“…Empirical data of the arm in use was also analysed. A full account of this study is presented by Blandford et al [1]. In brief, the analysis showed that EMU occupies a useful niche in the repertoire of evaluation techniques.…”