Usability Testing of a Digital Assessment Routing Tool: Protocol for an Iterative Convergent Mixed Methods Study

Lowe, Cabella; Sing, Harry Hanuman; Browne, Mitchell; Alwashmi, Meshari F; Marsh, William; Morrissey, Dylan

doi:10.2196/27205

Cited by 5 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The G-MoMo applications are efficient since users take less time to complete tasks accurately. This reconfirmed the results of earlier studies by (A'bas et al, 2021;Hussain and Omar, 2020;Lowe et al, 2021;Zakaria et al, 2020), where they found the applications efficient for the users. The users also agreed that they would remember navigating between G-MoMo applications pages the next time they use it.…”

Section: Discussionsupporting

confidence: 88%

“…The users achieved satisfaction with the features, functionalities, design, information and display quality of the G-MoMo applications. This finding was also reported in earlier studies by (A'bas et al, 2021;Al-Gayar et al, 2021;Lowe et al, 2021;Putri et al, 2021), where they observed that the users were satisfied with the applications' functionalities and design. The participants mentioned that the G-MoMo applications were easy to use, which helped them achieve satisfaction.…”

Section: Discussionsupporting

confidence: 88%

“…The G-MoMo applications were highly effective because they allowed mobile money subscribers to complete the tasks accurately to achieve their specified goals. This finding is logical to the studies by (A'bas et al, 2021;Byun et al, 2020;Lowe et al, 2021). The overall usability testing result is positive for the effectiveness of applications.…”

Section: Discussionmentioning

confidence: 75%

See 2 more Smart Citations

Heuristic Evaluation and Usability Testing of G-MoMo Applications

Ali

2022

J INFORM SYSTEMS ENG

View full text Add to dashboard Cite

Financial technology (FinTech) has swiftly revolutionized mobile money as one of the ways of accessing financial services in developing countries. Numerous mobile money applications were developed to access mobile money services but are hindered by severe authentication security challenges, thus, forcing the researchers to design a secure multi-factor authentication (MFA) algorithm for mobile money applications. Three prototypes of native mobile money applications (G-MoMo applications) were developed to confirm that the algorithm provides high security and is feasible. This study, therefore, aimed to evaluate the usability of the G-MoMo applications using heuristic evaluation and usability testing to identify potential usability issues and provide recommendations for improvement. Heuristic evaluation and usability testing methods were used to evaluate the G-MoMo applications. The heuristic evaluation was carried out by five experts that used the 10 principles proposed by Jakob Nielsen with a five-point severity rating scale to identify the usability problems. While the usability testing was conducted with forty participants selected using a purposive sampling method to validate the usability of the G-MoMo applications by performing tasks and filling out the post-test questionnaire. Data collected were analyzed in RStudio software. Sixty-three usability issues were identified during heuristic evaluation, where 33 were minor and 30 were major. The most violated heuristic items were "help and documentation", and "user control and freedom", while the least violated heuristic items were "aesthetic and minimalist design" and "visibility of system status". The usability testing findings revealed that the G-MoMo applications' performance proved good in learnability, effectiveness, efficiency, memorability, and errors. It also provided user satisfaction, ease of use, aesthetics, usefulness, integration, and understandability. Therefore, it was highly recommended that the developers of G-MoMo applications fix the identified usability problems to make the applications more reliable and increase overall user satisfaction.

show abstract

Section: Discussionsupporting

confidence: 88%

Section: Discussionsupporting

confidence: 88%

See 1 more Smart Citation

Heuristic Evaluation and Usability Testing of G-MoMo Applications

Ali

2022

J INFORM SYSTEMS ENG

View full text Add to dashboard Cite

show abstract

“…Real-world usability testing has been completed using an iterative convergent mixed methods design incorporating patient and public involvement [42], the results of which will be reported later in 2021. This study optimized usability in the final DART iteration that will be used in this pilot study and subsequent main trial.…”

Section: Previous Workmentioning

confidence: 99%

Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial

Lowe¹,

Sing²,

Marsh³

et al. 2021

JMIR Res Protoc

Self Cite

View full text Add to dashboard Cite

Background Musculoskeletal conditions account for 16% of global disability, resulting in a negative effect on millions of patients and an increasing demand for health care use. Digital technologies to improve health care outcomes and efficiency are considered a priority; however, innovations are rarely tested with sufficient rigor in clinical trials, which is the gold standard for clinical proof of safety and efficacy. We have developed a new musculoskeletal digital assessment routing tool (DART) that allows users to self-assess and be directed to the right care. DART requires validation in a real-world setting before implementation. Objective This pilot study aims to assess the feasibility of a future trial by exploring the key aspects of trial methodology, assessing the procedures, and collecting exploratory data to inform the design of a definitive randomized crossover noninferiority trial to assess DART safety and effectiveness. Methods We will collect data from 76 adults with a musculoskeletal condition presenting to general practitioners within a National Health Service (NHS) in England. Participants will complete both a DART assessment and a physiotherapist-led triage, with the order determined by randomization. The primary analysis will involve an absolute agreement intraclass correlation (A,1) estimate with 95% CI between DART and the clinician for assessment outcomes signposting to condition management pathways. Data will be collected to allow the analysis of participant recruitment and retention, randomization, allocation concealment, blinding, data collection process, and bias. In addition, the impact of trial burden and potential barriers to intervention delivery will be considered. The DART user satisfaction will be measured using the system usability scale. Results A UK NHS ethics submission was done during June 2021 and is pending approval; recruitment will commence in early 2022, with data collection anticipated to last for 3 months. The results will be reported in a follow-up paper in 2022. Conclusions This study will inform the design of a randomized controlled crossover noninferiority study that will provide evidence concerning mobile health DART system clinical signposting in an NHS setting before real-world implementation. Success should produce evidence of a safe, effective system with good usability, potentially facilitating quicker and easier patient access to appropriate care while reducing the burden on primary and secondary care musculoskeletal services. This rigorous approach to mobile health system testing could be used as a guide for other developers of similar applications. Trial Registration ClinicalTrials.gov NCT04904029; http://clinicaltrials.gov/ct2/show/NCT04904029 International Registered Report Identifier (IRRID) PRR1-10.2196/31541

show abstract

“…An iterative, convergent mixed methods design was used to assess the usability of DART, using cyclical evaluation and improvement plus mixed methods to provide richness while quantifying use, maximizing usability, and therefore supporting system adoption [ 34 ]. The testing protocol for this study has been described in detail in a previous publication [ 35 ].…”

Section: Introductionmentioning

confidence: 99%

Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: Iterative, Convergent Mixed Methods Study

Lowe¹,

Browne²,

Marsh³

et al. 2022

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

Background Musculoskeletal disorders negatively affect millions of patients worldwide, placing significant demand on health care systems. Digital technologies that improve clinical outcomes and efficiency across the care pathway are development priorities. We developed the musculoskeletal Digital Assessment Routing Tool (DART) to enable self-assessment and immediate direction to the right care. Objective We aimed to assess and resolve all serious DART usability issues to create a positive user experience and enhance system adoption before conducting randomized controlled trials for the integration of DART into musculoskeletal management pathways. Methods An iterative, convergent mixed methods design was used, with 22 adult participants assessing 50 different clinical presentations over 5 testing rounds across 4 DART iterations. Participants were recruited using purposive sampling, with quotas for age, habitual internet use, and English-language ability. Quantitative data collection was defined by the constructs within the International Organization for Standardization 9241-210-2019 standard, with user satisfaction measured by the System Usability Scale. Study end points were resolution of all grade 1 and 2 usability problems and a mean System Usability Scale score of ≥80 across a minimum of 3 user group sessions. Results All participants (mean age 48.6, SD 15.2; range 20-77 years) completed the study. Every assessment resulted in a recommendation with no DART system errors and a mean completion time of 5.2 (SD 4.44, range 1-18) minutes. Usability problems were reduced from 12 to 0, with trust and intention to act improving during the study. The relationship between eHealth literacy and age, as explored with a scatter plot and calculation of the Pearson correlation coefficient, was performed for all participants (r=–0.2; 20/22, 91%) and repeated with a potential outlier removed (r=–0.23), with no meaningful relationships observed or found for either. The mean satisfaction for daily internet users was highest (19/22, 86%; mean 86.5, SD 4.48; 90% confidence level [CL] 1.78 or –1.78), with nonnative English speakers (6/22, 27%; mean 78.1, SD 4.60; 90% CL 3.79 or –3.79) and infrequent internet users scoring the lowest (3/22, 14%; mean 70.8, SD 5.44; 90% CL 9.17 or –9.17), although the CIs overlap. The mean score across all groups was 84.3 (SD 4.67), corresponding to an excellent system, with qualitative data from all participants confirming that DART was simple to use. Conclusions All serious DART usability issues were resolved, and a good level of satisfaction, trust, and willingness to act on the DART recommendation was demonstrated, thus allowing progression to randomized controlled trials that assess safety and effectiveness against usual care comparators. The iterative, convergent mixed methods design proved highly effective in fully evaluating DART from a user perspective and could provide a blueprint for other researchers of mobile health systems. International Registered Report Identifier (IRRID) RR2-10.2196/27205

show abstract

Usability Testing of a Digital Assessment Routing Tool: Protocol for an Iterative Convergent Mixed Methods Study

Cited by 5 publications

References 24 publications

Heuristic Evaluation and Usability Testing of G-MoMo Applications

Heuristic Evaluation and Usability Testing of G-MoMo Applications

Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial

Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: Iterative, Convergent Mixed Methods Study

Contact Info

Product

Resources

About