Research on heuristic evaluation in recent years has focused on improving its effectiveness and efficiency with respect to user testing. The aim of this paper is to refine a research agenda for comparing and contrasting evaluation methods. To reach this goal, a framework is presented to evaluate the effectiveness of different types of support for structured usability problem reporting. This paper reports on an empirical study of this framework that compares two sets of heuristics, Nielsen's heuristics and the cognitive principles of Gerhardt-Powals, and two media of reporting a usability problem, i.e. either using a web tool or paper. The study found that there were no significant differences between any of the four groups in effectiveness, efficiency and inter-evaluator reliability. A more significant contribution of this research is that the framework used for the experiments proved successful and should be reusable by other researchers because of its thorough structure.

No abstract

Effectiveness of Heuristic Evaluation (HE) has severely been challenged. The present study investigated two strategies for improving the effectiveness of HE: Selection of usability guidelines (Gerhardt-Powals' cognitive engineering principles vs. Nielsen's heuristics); Provision of support and training. We also examined various strategies for estimating the effectiveness of HE. An empirical study with a balanced within-and betweensubject 2x2 factorial design involving 18 novice evaluators was conducted. 290 non-consolidated usability problems (UPs) identified in 35 HE test sessions were evaluated against 88 actual UPs found in the user tests (n = 17) performed earlier. Nielsen's heuristics were proved to be more effective in enabling the evaluators to uncover significantly more UPs, though in general the effectiveness of HE was low. Implications for future work are drawn.

No abstract

No abstract