Existing defect classification schemes are mainly used to characterize software defects. A few of them are specifically applicable to usability problems, but they have not been validated and their reliability has been assessed in a limited way. The aim of this study is to evaluate comprehensively the Classification Usability Problems (CUP) scheme. First, the reliability was evaluated with raters of different levels of expertise and experience in using CUP. Second, the acceptability was assessed with a questionnaire. Third, the validity was assessed with developers in a field study. Results show that some form of training is required for inexperienced evaluators to exploit CUP fully, but a simplified version of CUP may still be useful for developers and usability practitioners. The evaluation framework employed proved effective for revising CUP and may be applied to validate other related schemes.

The aim of this study was to evaluate the Classification of Usability Problems (CUP) scheme. The goal of CUP is to classify usability problems further to give user interface developers better feedback to improve their understanding of usability problems, help them manage usability maintenance, enable them to find effective fixes for UP, and prevent such problems from reoccurring in the future. First, reliability was evaluated with raters of different levels of expertise and experience in using CUP. Second, acceptability was assessed with a questionnaire. Third, validity was assessed by developers in two field studies. An analytical comparison was also made to three other classification schemes. CUP reliability results indicated that the expertise and experience of raters are critical factors for assessing reliability consistently, especially for the more complex attributes. Validity analysis results showed that tools used by developers must be tailored to their working framework, knowledge and maturity. The acceptability study showed that practitioners are concerned with the effort spent in applying any tool. To understand developers' work and the implications of this study two theories are presented for understanding and prioritising UP. For applying classification schemes, the implications of this study are that training and context are needed.