Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides

Abstract

Background There is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed. Objective The main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders. Methods In order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability. Results Only a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (α=0.78) and “Privacy” (α=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29. Conclusions This study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

Mental, Physical and Socio-Economic Status of Adults Living in Spain during the Late Stages of the State of Emergency Caused by COVID-19

“…In relation to this particular issue, mobile health-related applications have been suggested as an alternative to provide easy, immediate access to the treatments that are needed [ 29 ]. However, not all available health-related mobile apps have been developed on the basis of scientific tenets [ 30 ] following available guides, and undergoing usability tests and efficacy studies [ 31 ]. Therefore, additional studies are needed to improve the development of data-based health-related mobile apps and facilitate a wider implementation and use among healthcare professionals and patients.…”

Quality assessment of mHealth apps: a scoping review

“…In 15 of the included articles, a quality assurance system or a quality assessment tool was developed (10,21,(28)(29)(30)(31)(32)(33)(34)(35)(36)(37)(38)(39)(40). Five of the included articles were validation studies (8,(41)(42)(43)(44). In addition to development and validation studies, a number of studies (n = 39) were identified in which quality assessment tools were applied.…”

Effects of COVID-19 Social Distancing Measures in Individuals with Chronic Pain Living in Spain in the Late Stages of the Lockdown

“…In relation to this particular issue, the so-called new technologies (e.g., mobile apps) have been suggested as an alternative [41] although not all pain-related mobile apps have been developed on the basis of scientific tenets [42], and most have not been tested before dissemination [43]. Therefore, not all can be used safely [44]. However, studies show that mobile apps can effectively help to measure pain, (e.g., [30]), and self-manage different pain conditions, such as back pain [45], fibromyalgia [46] or cancer-related pain [47].…”