A framework for human factors evaluation

Whitefield, Andy; Wilson, Frank E.; Dowell, J. D.

doi:10.1080/01449299108924272

Cited by 98 publications

(61 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In particular, sampling the objects that are present in the real-life environment and that are relevant for users to achieve their goals seems to be more important for task representativeness than the representativeness of the prototype. This finding is consistent with earlier findings about representative design (Whitefield, Wilson, & Dowell, 1991;Sefelin, Tscheligi, & Giller, 2003).…”

Section: Summary Of the Resultssupporting

confidence: 83%

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

Kieffer¹

2017

THCI

View full text Add to dashboard Cite

Abstract:Egon Brunswik coined and defined the concepts of ecological validity and representative design, which are both essential to achieve external validity. However, research in HCI has inconsistently and incorrectly used Brunswik's concept of ecological validity, which prevents the field from developing cumulative science and from generalizing the findings of user experience (UX) evaluations. In this paper, I present ECOVAL, a framework I built on Brunswik's ideas. On the one hand, ECOVAL helps HCI researchers describe and assess the ecological validity of cues in UX evaluations. On the other hand, ECOVAL guidelines-formulated as a step-by-step procedure-help HCI researchers achieve representative design and, therefore, increase external validity. An industrial case study demonstrates the relevance of ECOVAL for achieving representative design while conducting formative UX testing. In discussing the case study, I describe how ECOVAL can help HCI researchers assess and increase the validity of UX experiments and generalize UX findings. I also illustrate the trade-offs between internal and external validities and UX resources that inevitably arise when one conducts UX experiments. From the results, I sketch avenues for future research and discuss the related challenges that future work should address.

show abstract

Section: Summary Of the Resultssupporting

confidence: 83%

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

Kieffer¹

2017

THCI

View full text Add to dashboard Cite

show abstract

“…Whitefield, Wilson, and Dowell (1991) suggest that evaluation in the human factors discipline involves an assessment of the conformity between a system's performance and its desired performance. In many cases, practitioners may assume that evaluation is a single event that occurs at or near the end of the design process.…”

Section: Usability Evaluation Methodsmentioning

confidence: 99%

“…The five dimensions include: (1) formative versus summative, (2) discovery versus decision, (3) formalized versus informal, (4) designer involvement versus user involvement, and (5) complete and component. Whitefield, Wilson, and Dowell (1991) provided a framework for human factors evaluation using four categories: (1) analytic methods, (2) specialist reports, (3) user reports, and (4) …”

Section: Usability Evaluation Methodsmentioning

confidence: 99%

“…Definitions frequently vary as to what evaluation really means, and are sometimes discipline specific. In military systems, evaluation infers testing system performance against operational requirements.For many commercial systems, evaluation may refer to conformance with design specifications.Whitefield, Wilson, and Dowell (1991) suggest that evaluation in the human factors discipline involves an assessment of the conformity between a system's performance and its desired performance. In many cases, practitioners may assume that evaluation is a single event that occurs at or near the end of the design process.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems

2003

View full text Add to dashboard Cite

The need for cost-effective usability evaluation has led to the development of methodologies to support the usability practitioner in finding usability problems during formative evaluation. Even though various methods exist for performing usability evaluation, practitioners seldom have the information needed to decide which method is appropriate for their specific purpose. In addition, most methods do not have an integrated relationship with a theoretical foundation for applying the method in a reliable and efficient manner. Practitioners often have to apply their own judgment and techniques, leading to inconsistencies in how the method is applied in the field. Usability practitioners need validated information to determine if a given usability evaluation method is effective and why it should be used instead of some other method. Such a desire motivates the need for formal, empirical comparison studies to evaluate and compare usability evaluation methods. In reality, the current data for comparing usability evaluation methods suffers from a lack of consistent measures, standards, and criteria for identifying effective methods.The work described here addresses three important research activities. First, the User Action Framework was developed to help organize usability concepts and issues into a knowledge base that supports usability methods and tools. From the User Action Framework, a mapping was made to the Usability Problem Inspector; a tool to help practitioners conduct a highly focused inspection of an interface design. Second, the reliability of the User Action Framework was evaluated to determine if usability practitioners could use the framework in a consistent manner when classifying a set of usability problems. Third, a comprehensive comparison study was conducted to determine if the Usability Problem Inspector, based on the User Action Framework, could produce results just as effective as two other inspection methods (i.e., the heuristic evaluation and the cognitive walkthrough). The comparison study used a new comparison approach with standards, measures, and criteria to prove the effectiveness of methods. Results from the User Action Framework reliability study showed higher agreement scores at all classification levels than was found in previous work with a similar classification iii tool. In addition, agreement using the User Action Framework was stronger than the results obtained from the same experts using the heuristic evaluation. From the inspection method comparison study, results showed the Usability Problem Inspector to be more effective than the heuristic evaluation and consistent with effectiveness scores from the cognitive walkthrough.iv ACKNOWLEDGMENTS I would like to take this opportunity to gratefully acknowledge many individuals that helped make my graduate experience very rewarding. First, I am indebted to my wife (Debbie) and my two children (Jennifer and Brian). Graduate school can be a very lonely endeavor, but I am so thankful I was able to share the experience with my family....

show abstract

“…Nielsen [8] stated, as a first rule of usability, "don't listen to users" and argued that users' design feedback should be limited to preference data after having used the interactive system in question. Users' design feedback may be biased due to a desire to report what the evaluator wants to hear, imperfect memory, and rationalization of own behaviour [8,9]. As discussed by Gould and Lewis [10], it can be challenging to elicit useful design information from users as they may not have considered alternative approaches or may be ignorant of relevant alternatives; users may simply be unaware of what they need.…”

Section: Researchmentioning

confidence: 99%

Users’ design feedback in usability evaluation: a literature review

Følstad

2017

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

IntroductionInvolving users in usability evaluation is valuable when designing information and communication technology (ICT), and a range of usability evaluation methods (UEM) support user involvement. Relevant methods include adaptations of usability testing [1], usability inspection methods such as pluralistic walkthrough [2], and inquiry methods such as interviews [3], and focus groups [4].Users involved in usability evaluation may generate two types of data. We term these interaction data and design feedback. Interaction data are recordings of the actual use of an interactive system, such as observational data, system logs, and data from think-aloud protocols. Design feedback are data on users' reflections concerning an interactive system, such as comments on experiential issues, considerations of the system's suitability for its context of use, usability problem predictions, and design suggestions. AbstractAs part of usability evaluation, users may be invited to offer their reflections on the system being evaluated. Such reflections may concern the system's suitability for its context of use, usability problem predictions, and design suggestions. We term the data resulting from such reflections users' design feedback. Gathering users' design feedback as part of usability evaluation may be seen as controversial, and the current knowledge on users' design feedback is fragmented. To mitigate this, we have conducted a literature review. The review provides an overview of the benefits and limitations of users' design feedback in usability evaluations. Following an extensive search process, 31 research papers were identified as relevant and analysed. Users' design feedback is gathered for a number of distinct purposes: to support budget approaches to usability testing, to expand on interaction data from usability testing, to provide insight into usability problems in users' everyday context, and to benefit from users' knowledge and creativity. Evaluation findings based on users' design feedback can be qualitatively different from, and hence complement, findings based on other types of evaluation data. Furthermore, findings based on users' design feedback can hold acceptable validity, though the thoroughness of such findings may be questioned. Finally, findings from users' design feedback may have substantial impact in the downstream development process. Four practical implications are highlighted, and three directions for future research are suggested. RESEARCHFølstad Hum. Cent. Comput. Inf. Sci. (2017) Cent. Comput. Inf. Sci. (2017) 7:19 The value of interaction data in evaluation is unchallenged. Interaction data is held to be a key source of insight in the usability of interactive systems and has been the object of thorough scientific research. Numerous empirical studies concern the identification of usability problems on the basis of observable user behaviour [5]. Indeed, empirical UEM assessments are typically done by comparing the set of usability problems identified through the assessed UEM with a ...

show abstract

A framework for human factors evaluation

Cited by 98 publications

References 10 publications

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

ECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations

Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems

Users’ design feedback in usability evaluation: a literature review

Contact Info

Product

Resources

About