What do usability evaluators do in practice?

Nørgaard, Mie; Hornbæk, Kasper

doi:10.1145/1142405.1142439

Cited by 101 publications

(21 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conducted the study in a separate session with each subject. At the beginning of each session, we asked the participants to 'think aloud' [47,50,81] during the study. Specifically, we asked them to verbally describe what they were doing, to comment on any of their concerns, and to say whatever comes to their mind while solving the given tasks.…”

Section: Study Process and Data Collectionmentioning

confidence: 99%

Evaluating an Interactive Memory Analysis Tool: Findings from a Cognitive Walkthrough and a User Study

Weninger

Grünbacher

Gander

et al. 2020

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Memory analysis tools are essential for finding and fixing anomalies in the memory usage of software systems (e.g., memory leaks). Although numerous tools are available, hardly any empirical studies exist on their usefulness for developers in typical usage scenarios. Instead, most evaluations are limited to reporting performance metrics. We thus conducted a study to empirically assess the usefulness of the interactive memory analysis tool AntTracks Analyzer. Specifically, we first report findings from assessing the tool using a cognitive walkthrough, guided by the Cognitive Dimensions of Notations Framework. We then present the results of a qualitative user study involving 14 subjects who used AntTracks to detect and resolve memory anomalies. We report lessons learned from the study and implications for developers of interactive memory analysis tools. We hope that our results will help researchers and developers of memory analysis tools in defining, selecting, and improving tool capabilities.

show abstract

Section: Study Process and Data Collectionmentioning

confidence: 99%

Evaluating an Interactive Memory Analysis Tool: Findings from a Cognitive Walkthrough and a User Study

Weninger

Grünbacher

Gander

et al. 2020

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

show abstract

“…Several code improvements techniques and protocols have been used for AVISPA refactoring. The think-aloud method [13] has been used in order to retrieve software user experience, AVISPA was assessed with this methodology as an exploratory research of tool inner construction, code concepts and architectural decisions. Also, allows to retrieve the software process models experts' knowledge and opinion.…”

Section: Avispa Code Improvement Techniquesmentioning

confidence: 99%

An experimental design on the SPEM 2.0 process model element classification algorithm of the AVISPA tool through ANOVA variance analysis

Londoño

Alegria

2020

ing. Solidar

View full text Add to dashboard Cite

Introduction: This publication is the product of research: “SPEM 2.0 Process Model Metrics in the Reliability of its Visual Analysis” throughout 2019, which supports the work of a master’s degree in Systems Engineering at the University of Cauca. Objective: Rebase a process model metrics set in order to increase AVISPA reliability to support the visual analysis of SPEM 2.0 software process models. Methodology: In order to improve AVISPA, a systematic literature review had been performed to find software process model metrics that are potentially apt to be represented in AVISPA. Next, a set of assessments were performed in order to enhance visual analysis tool. Finally, an ANOVA statistical assessment was realized in order to find a variance differential between AVISPA versions by comparing their F1-Score process model elements classification values. Results: AVISPA significantly improved its general classification algorithm. Most of errors were found in SPEM 2.0 variability resolution feature and collections with duplicated elements. Multiple misclassifications still persists. Conclusion: General AVISPA process model elements classification is improved. However, some process model samples remain scattered according to ANOVA results. Originality: AVISPA is a recent solution for SPEM 2.0 software process model assessment. It's recent emergence carried to a lack of articles about software process model metrics and few works about AVISPA improvements. These are the main contributions of this paper. Limitations: The project has been widely expensive in terms of execution time, traceability with all software process model elements, and mainly to find experts in software process that can meet the research requirements

show abstract

“…It is also a regular topic in practitioner discourse [87,64]. Equally important and perhaps more extensive has been the topic of 'thinkaloud' protocols and their implementation in testing [53,10,73,6,15,52], along with some interest in the divergences between academic and practitioner uses of such techniques [67]. Boren and Ramey [6] in particular offer one of the most detailed accounts of what testing involves from the perspective of moderator-participant interactions.…”

Section: Lab-based Usability Testing In Hci Research and Ux Practicementioning

confidence: 99%

“…For instance, Boren and Ramey argue for the theoretical grounding of usability testing practices, pointing to deficiencies in the conduct of usability practitioners in moderating tests [6]. Nørgaard and Hornbaek, who do not seek to "reprehend the practice of usability testing", nevertheless still sound a note of concern over practitioners' work regarding absences of systematic analysis and potential bias 3 (e.g., in the anticipation of usability problems and their shaping the design of test tasks) [73]. In some ways it is unsurprising that this framing is applied to industrial applications of usability testing, since a significant portion of HCI research also treats usability testing performed in academic research environments in a normative scientific frame: one need only refer to the well-known "damaged merchandise" discussions of the 1990s [44,74] in which this orientation to 'usability as normal science' has been litigated (and the confusions around "validat[ing] discount methods" [16]).…”

Section: Conceptualising Usability Testingmentioning

confidence: 99%

How UX Practitioners Produce Findings in Usability Testing

Reeves

2019

ACM Trans. Comput.-Hum. Interact.

View full text Add to dashboard Cite

Usability testing has long been a core interest of HCI research and forms a key element of industry practice. Yet our knowledge of it harbours striking absences. There are few, if any detailed accounts of the contingent, material ways in which usability testing is actually practiced . Further, it is rare that industry practitioners’ testing work is treated as indigenous and particular (instead subordinated as a ‘compromised’ version). To service these problems, this article presents an ethnomethodological study of usability testing practices in a design consultancy. It unpacks how findings are produced in and as the work of observers analysing the test as it unfolds between moderators taking participants through relevant tasks. The study nuances conventional views of usability findings as straightforwardly ‘there to be found’ or ‘read off’ by competent evaluators. It explores how evaluators/observers collaboratively work to locate relevant troubles in the test's unfolding. However, in the course of doing this work, potential candidate troubles may also routinely be dissipated and effectively ‘ignored’ in one way or another. The implications of the study suggest refinements to current understandings of usability evaluations, and affirm the value to HCI in studying industry practitioners more deeply.

show abstract

What do usability evaluators do in practice?

Cited by 101 publications

References 26 publications

Evaluating an Interactive Memory Analysis Tool: Findings from a Cognitive Walkthrough and a User Study

Evaluating an Interactive Memory Analysis Tool: Findings from a Cognitive Walkthrough and a User Study

An experimental design on the SPEM 2.0 process model element classification algorithm of the AVISPA tool through ANOVA variance analysis

How UX Practitioners Produce Findings in Usability Testing

Contact Info

Product

Resources

About