From expert-derived user needs to user-perceived ease of use and usefulness: A two-phase mixed-methods evaluation framework

Abstract

Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of “the best available” comparison system to enable a cognitive walkthrough in phase 1 and a comparative effectiveness evaluation in phase 2. During phase 2, five quantitative and qualitative evaluation methods are mixed to assess usability: time-motion analysis, software log, questionnaires – System Usability Scale and the Unified Theory of Acceptance of Use of Technology, think-aloud protocols, and unstructured interviews. Each method contributes data for a unique measure (e.g., time motion analysis contributes task-completion-time; software log contributes action transition frequency). The measures are triangulated to yield complementary insights regarding user-perceived ease-of-use, functionality integration, anxiety during use, and workflow impact. To illustrate its use, we applied this framework in a formative evaluation of a software called Integrated Model for Patient Care and Clinical Trials (IMPACT). We conclude that this mixed-methods evaluation framework enables an integrated assessment of user needs satisfaction and user-perceived usefulness and usability of a novel design. This evaluation framework effectively bridges the gap between co-evolving user needs and technology designs during iterative prototyping and is particularly useful when it is difficult for users to articulate their needs for technology support due to the lack of a baseline.

Bringing clinical pharmacogenomics information to pharmacists: A qualitative study of information needs and resource requirements

“…A laboratory-based task analysis[38,43] will assess the utility of the tool in helping pharmacists locate and use information to make clinical decisions.…”

Biofeedback in Partial Weight Bearing: Usability of Two Different Devices from a Patient’s and Physical Therapist’s Perspective

“…In this framework several qualitative and quantitative evaluation methods have been proposed. Mixing qualitative and quantitative evaluation methods ensures comprehensive data collection and avoids needless a priori assumptions [26,34,36]. However, there are also limitations of using the conceptual framework of De Bleser et al This framework is not widely adopted in usability testing and although it builds on existing literature and frameworks it remains unclear how valid this conceptual framework itself is.…”

“…User performance was studied using three methods: (1) counting errors of use during the performed tasks; (2) timing user tasks in seconds; (3) asking participants to think aloud while performing tasks. A think-aloud method is commonly used in usability research, especially when users are confronted for the first time with a device [26,34–36]. This method is used to make explicit what users thoughts and experiences are when performing a specific task.…”

Usability of a store and forward telehealth platform for diagnosis and management of oral mucosal lesions: a cross-sectional study (Preprint)

“…During this simulation, the participants were guided to "think out loud," that is, to express their feelings and difficulties while performing the task. In usability research, this approach has been useful to identify problems in information systems [14,15].…”