Challenges of Traditional Usability Evaluation in End-User Development

Abstract

End-user development (EUD) research has yielded a variety of novel environments and techniques, often accompanied by lab-based usability studies that test their effectiveness in the completion of representative real-world tasks. While lab studies play an important role in resolving frustrations and demonstrating the potential of novel tools, they are insufficient to accurately determine the acceptance of a technology in its intended context of use, which is highly dependent on the diverse and dynamic requirements of its users, as we show here. As such, usability in the lab is unlikely to represent usability in the field. To demonstrate this, we first describe the results of a think-aloud usability study of our EUD tool "Jeeves", followed by two case studies where Jeeves was used by psychologists in their work practices. Common issues in the artificial setting were seldom encountered in the real context of use, which instead unearthed new usability issues through unanticipated user needs. We conclude with considerations for usability evaluation of EUD tools that enable development of software for other users, including planning for collaborative activities, supporting developers to evaluate their own tools, and incorporating longitudinal methods of evaluation.

End-User Development of Experience Sampling Smartphone Apps -Recommendations and Requirements

“…Our previous work described Jeeves in relation to EUD adoption requirements for psychology researchers [21], as well as how usability issues in a lab study of Jeeves were almost entirely separate to real-world usability challenges identified in case studies [22]. In this work, we extend on both of these results by triangulating interview feedback with analysis of our case studies.…”

“…As previously discussed, interviews alone were insufficient to determine challenges pertaining to real-world use of EUD-ESM. This section thus describes three applications of Jeeves by psychology researchers, two of which were previously discussed in prior work [22] and one currently in progress. In doing so, we highlight issues that have emerged in practical use that provide concrete suggestions for implementing requirements identified in interviews.…”

“…In response, we have developed the Jeeves tool to allow end-users with no programming skills to create their own ESM apps, couple them to participants' smartphones, and receive assessment results in real-time. While our previous work has addressed design decisions and lab-based usability studies of Jeeves [20,22], this paper extends the evaluation conducted in [21] to synthesise necessary functionality and constraints of EUD for ESM apps (hereby referred to as EUD-ESM) addressing the following question:…”

A Novel Experience Sampling Method Tool Integrating Momentary Assessments of Cognitive Biases: Two Compliance, Usability, and Measurement Reactivity Studies

“…This is particularly relevant because of the development of a new system that integrates an app-based cognitive task into the ESM procedure. Previous platforms have been developed to repeatedly assess affect through ESM, provide a visual environment [ 55 - 57 ], and highlight the importance of usability testing. In the case of the new ESM system used in our studies, the results on usability showed a very high level of learnability (ie, over the 90th percentile of the SUS), which reflects the ease of use and acceptability by the participants.…”

DRAFT-What you always wanted to know but could not find about block-based environments

“…This evaluation was done using the cognitive dimensions of notations framework [21]. Likewise, Rough and Quigley [126] present the challenges of traditional usability evaluations. Almjally et al [4] present an empirical study that compares the usage of a block-based language using Tangible User Interfaces (TUIs) and Graphical User Interfaces (GUIs).…”