Conceptualizing Usability for the eHealth Context: Content Analysis of Usability Problems of eHealth Applications

Abstract

Background Usability tests can be either formative (where the aim is to detect usability problems) or summative (where the aim is to benchmark usability). There are ample formative methods that consider user characteristics and contexts (ie, cognitive walkthroughs, interviews, and verbal protocols). This is especially valuable for eHealth applications, as health conditions can influence user-system interactions. However, most summative usability tests do not consider eHealth-specific factors that could potentially affect the usability of a system. One of the reasons for this is the lack of fine-grained frameworks or models of usability factors that are unique to the eHealth domain. Objective In this study, we aim to develop an ontology of usability problems, specifically for eHealth applications, with patients as primary end users. Methods We analyzed 8 data sets containing the results of 8 formative usability tests for eHealth applications. These data sets contained 400 usability problems that could be used for analysis. Both inductive and deductive coding were used to create an ontology from 6 data sets, and 2 data sets were used to validate the framework by assessing the intercoder agreement. Results We identified 8 main categories of usability factors, including basic system performance, task-technology fit, accessibility, interface design, navigation and structure, information and terminology, guidance and support, and satisfaction. These 8 categories contained a total of 21 factors: 14 general usability factors and 7 eHealth-specific factors. Cohen κ was calculated for 2 data sets on both the category and factor levels, and all Cohen κ values were between 0.62 and 0.67, which is acceptable. Descriptive analysis revealed that approximately 69.5% (278/400) of the usability problems can be considered as general usability factors and 30.5% (122/400) as eHealth-specific usability factors. Conclusions Our ontology provides a detailed overview of the usability factors for eHealth applications. Current usability benchmarking instruments include only a subset of the factors that emerged from our study and are therefore not fully suited for summative evaluations of eHealth applications. Our findings support the development of new usability benchmarking tools for the eHealth domain.

Improving usability benchmarking for the eHealth domain: The development of the eHealth UsaBility Benchmarking instrument (HUBBI)

“…Additionally, the HUBBI has some overlap in terms of measurement items in other instruments, like the PSSUQ ( The system gave error messages that clearly told me how to fix problems ), MAUQ ( overall , I am satisfied with this system ) and Health-ITUES ( the information provided with [system] is clear) . But while each of these questionnaires contain some elements of the HUBBI, they do not evaluate the full scope of eHealth usability , as covered in the ontology for eHealth usability [ 15 ]. In this ontology, 70% of the factors are general usability factors that are relevant to all digital technologies regardless of their specific domain.…”

“…The other 30% are eHealth-specific factors that are essential for evaluating usability of eHealth applications. Likewise, in the work by Broekhuis et al [ 15 ] an analysis of the usability issues identified within eight datasets was presented in which the division of general usability issues versus eHealth-specific usability issues (based on the ontology mentioned before) also displayed this 70%-30% split. The HUBBI reflects this 70%-30% division, as five out of the 18 items are eHealth-specific items while the remaining items are more generally formulated.…”

“…When looking at other usability questionnaires, it shows that the PSSUQ does not include any health-related items, the MAUQ does not include technical performance of the application nor the understandability of (medical) information in the app and the Health-ITUES does not include items on the fit between system and user, context or health goals nor items related to interface design. From the study of Broekhuis et al [ 15 ] it became clear that these topics are of importance to evaluate the usability of eHealth services. The HUBBI, that is based on this ontology of eHealth usability, includes both those general and eHealth-specific usability factors that directly affect user interaction with an eHealth service.…”

“…While the emergence of eHealth-specific instruments is a very positive development, the community seems to have skipped an important step: creating a comprehensive understanding of the usability concept for the eHealth context. A recent study proposed a new ontology of usability factors, specifically for the eHealth domain [ 15 ], based on a content analysis of 400 usability problems found in eight contemporary eHealth services. These eHealth services cover a wide range of different systems like a gamified exercise module, home monitoring tool, a robotic screening tool, mobile apps and an online coaching platform.…”

“…The ontology takes both into account, identifying both general and eHealth-specific factors that need to be considered when evaluating the usability of eHealth services. The study by Broekhuis et al [ 15 ] further shows how 30% of the usability problems in eHealth services are related to eHealth-specific factors. This means that current, generic usability benchmarking instruments, such as the SUS [ 2 ] and the PSSUQ [ 4 ] measure only a subset (70% at best) of the general usability factors and ignore eHealth-specific usability factors.…”