A Scale to Assess the Methodological Quality of Studies Assessing Usability of Electronic Health Products and Services: Delphi Study Followed by Validity and Reliability Testing

Abstract

BackgroundThe usability of electronic health (eHealth) and mobile health apps is of paramount importance as it impacts the quality of care. Methodological quality assessment is a common practice in the field of health for different designs and types of studies. However, we were unable to find a scale to assess the methodological quality of studies on the usability of eHealth products or services.ObjectiveThis study aimed to develop a scale to assess the methodological quality of studies assessing usability of mobile apps and to perform a preliminary analysis of of the scale’s feasibility, reliability, and construct validity on studies assessing usability of mobile apps, measuring aspects of physical activity.MethodsA 3-round Delphi panel was used to generate a pool of items considered important when assessing the quality of studies on the usability of mobile apps. These items were used to write the scale and the guide to assist its use. The scale was then used to assess the quality of studies on usability of mobile apps for physical activity, and it assessed in terms of feasibility, interrater reliability, and construct validity.ResultsA total of 25 experts participated in the Delphi panel, and a 15-item scale was developed. This scale was shown to be feasible (time of application mean 13.10 [SD 2.59] min), reliable (intraclass correlation coefficient=0.81; 95% CI 0.55-0.93), and able to discriminate between low- and high-quality studies (high quality: mean 9.22 [SD 0.36]; low quality: mean 6.86 [SD 0.80]; P=.01).ConclusionsThe scale that was developed can be used both to assess the methodological quality of usability studies and to inform its planning.

Methodological Quality of Manuscripts Reporting on the Usability of Mobile Applications for Pain Assessment and Management: A Systematic Review

“…The methodological quality of included studies was independently assessed by two reviewers (AGS and AFA) using the Critical Assessment of Usability Studies Scale [23]. This is a recently developed scale, which is both valid and reliable (Intraclass Correlation Coefficient-ICC = 0.81) and scores vary between 0 and 100% [23].…”

“…The methodological quality of included studies was independently assessed by two reviewers (AGS and AFA) using the Critical Assessment of Usability Studies Scale [23]. This is a recently developed scale, which is both valid and reliable (Intraclass Correlation Coefficient-ICC = 0.81) and scores vary between 0 and 100% [23]. This scale is composed of fifteen questions on the procedures used to assess usability: 1) Did the study use valid measurement instruments of usability (i.e., there is evidence that the instruments used assess usability)?…”

“…A recent systematic review attempted to assess the quality of usability studies using a four-point scale [22] but failed to report on the process of development of the scale, or its reliability or validity, fundamental characteristics of any assessment or measurement instrument. Recently, a 15-items scale was developed to assess the methodological quality of usability studies for eHealth applications or general applications [23]. This scale was developed using a robust process, including a three-round Delphi method with 25 usability experts to generate a first version of the scale, which was then tested for comprehensibility by another three experts to generate a final version of the scale.…”

“…This scale was developed using a robust process, including a three-round Delphi method with 25 usability experts to generate a first version of the scale, which was then tested for comprehensibility by another three experts to generate a final version of the scale. This final version was shown to be feasible and both valid and reliable [23]. The present review aims to synthesize and evaluate existing studies on the assessment of the usability of pain-related apps using this newly developed scale [23].…”

“…This final version was shown to be feasible and both valid and reliable [23]. The present review aims to synthesize and evaluate existing studies on the assessment of the usability of pain-related apps using this newly developed scale [23].…”