Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial

Lowe, Cabella; Sing, Harry Hanuman; Marsh, William; Morrissey, Dylan

doi:10.2196/31541

Cited by 2 publications

(2 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The purpose of this usability study was to optimize usability before proceeding to a trial evaluating the safety and effectiveness of DART against a usual care comparator. A protocol for an initial pilot study has been published and will explore the key aspects of the trial methodology; assess the procedures; and collect exploratory data to inform the design of a definitive, randomized, crossover, noninferiority trial to assess DART safety [ 36 ]. DART is currently deployed in a controlled live clinical environment where we use system data, as well as user and clinician feedback, to further refine the algorithms and system usability.…”

Section: Discussionmentioning

confidence: 99%

“…The protocol devised for this usability study went through a series of iterations within an internal review process, comprising the research project team and DART system developers to arrive at the final version [ 35 ]. The objective of this study was to optimize usability before evaluating the safety and effectiveness of DART through a randomized controlled trial, the pilot protocol for which has been published [ 36 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: Iterative, Convergent Mixed Methods Study

Lowe¹,

Browne²,

Marsh³

et al. 2022

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

Background Musculoskeletal disorders negatively affect millions of patients worldwide, placing significant demand on health care systems. Digital technologies that improve clinical outcomes and efficiency across the care pathway are development priorities. We developed the musculoskeletal Digital Assessment Routing Tool (DART) to enable self-assessment and immediate direction to the right care. Objective We aimed to assess and resolve all serious DART usability issues to create a positive user experience and enhance system adoption before conducting randomized controlled trials for the integration of DART into musculoskeletal management pathways. Methods An iterative, convergent mixed methods design was used, with 22 adult participants assessing 50 different clinical presentations over 5 testing rounds across 4 DART iterations. Participants were recruited using purposive sampling, with quotas for age, habitual internet use, and English-language ability. Quantitative data collection was defined by the constructs within the International Organization for Standardization 9241-210-2019 standard, with user satisfaction measured by the System Usability Scale. Study end points were resolution of all grade 1 and 2 usability problems and a mean System Usability Scale score of ≥80 across a minimum of 3 user group sessions. Results All participants (mean age 48.6, SD 15.2; range 20-77 years) completed the study. Every assessment resulted in a recommendation with no DART system errors and a mean completion time of 5.2 (SD 4.44, range 1-18) minutes. Usability problems were reduced from 12 to 0, with trust and intention to act improving during the study. The relationship between eHealth literacy and age, as explored with a scatter plot and calculation of the Pearson correlation coefficient, was performed for all participants (r=–0.2; 20/22, 91%) and repeated with a potential outlier removed (r=–0.23), with no meaningful relationships observed or found for either. The mean satisfaction for daily internet users was highest (19/22, 86%; mean 86.5, SD 4.48; 90% confidence level [CL] 1.78 or –1.78), with nonnative English speakers (6/22, 27%; mean 78.1, SD 4.60; 90% CL 3.79 or –3.79) and infrequent internet users scoring the lowest (3/22, 14%; mean 70.8, SD 5.44; 90% CL 9.17 or –9.17), although the CIs overlap. The mean score across all groups was 84.3 (SD 4.67), corresponding to an excellent system, with qualitative data from all participants confirming that DART was simple to use. Conclusions All serious DART usability issues were resolved, and a good level of satisfaction, trust, and willingness to act on the DART recommendation was demonstrated, thus allowing progression to randomized controlled trials that assess safety and effectiveness against usual care comparators. The iterative, convergent mixed methods design proved highly effective in fully evaluating DART from a user perspective and could provide a blueprint for other researchers of mobile health systems. International Registered Report Identifier (IRRID) RR2-10.2196/27205

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: Iterative, Convergent Mixed Methods Study

Lowe¹,

Browne²,

Marsh³

et al. 2022

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

show abstract

Evaluation of a Musculoskeletal Digital Assessment Routing Tool (DART): Crossover Noninferiority Randomized Pilot Trial

Lowe,

Sephton,

Marsh

et al. 2024

JMIR Form Res

View full text Add to dashboard Cite

Background Musculoskeletal conditions account for 16% of global disability, resulting in a negative effect on patients and increasing demand for health care use. Triage directing patients to appropriate level intervention improving health outcomes and efficiency has been prioritized. We developed a musculoskeletal digital assessment routing tool (DART) mobile health (mHealth) system, which requires evaluation prior to implementation. Such innovations are rarely rigorously tested in clinical trials—considered the gold standard for evaluating safety and efficacy. This pilot study is a precursor to a trial assessing DART performance with a physiotherapist-led triage assessment. Objective The study aims to evaluate trial design, assess procedures, and collect exploratory data to establish the feasibility of delivering an adequately powered, definitive randomized trial, assessing DART safety and efficacy in an NHS primary care setting. Methods A crossover, noninferiority pilot trial using an integrated knowledge translation approach within a National Health Service England primary care setting. Participants were patients seeking assessment for a musculoskeletal condition, completing a DART assessment and the history-taking element of a face-to-face physiotherapist-led triage in a randomized order. The primary outcome was agreement between DART and physiotherapist triage recommendation. Data allowed analysis of participant recruitment and retention, randomization, blinding, study burden, and potential barriers to intervention delivery. Participant satisfaction was measured using the System Usability Scale. Results Over 8 weeks, 129 patients were invited to participate. Of these, 92% (119/129) proceeded to eligibility assessment, with 60% (78/129) meeting the inclusion criteria and being randomized into each intervention arm (39/39). There were no dropouts and data were analyzed for all 78 participants. Agreement between physiotherapist and DART across all participants and all primary triage outcomes was 41% (32/78; 95% CI 22-45), intraclass correlation coefficient 0.37 (95% CI 0.16-0.55), indicating that the reliability of DART was poor to moderate. Feedback from the clinical service team led to an adjusted analysis yielding of 78% (61/78; 95% CI 47-78) and an intraclass correlation coefficient of 0.57 (95% CI 0.40-0.70). Participant satisfaction was measured quantitively using amalgamated System Usability Scale scores (n=78; mean score 84.0; 90% CI +2.94 to –2.94), equating to an “excellent” system. There were no study incidents, and the trial burden was acceptable. Conclusions Physiotherapist-DART agreement of 78%, with no adverse triage decisions and high patient satisfaction, was sufficient to conclude DART had the potential to improve the musculoskeletal pathway. Study validity was enhanced by the recruitment of real-world patients and using an integrated knowledge translation approach. Completion of a context-specific consensus process is recommended to provide definitive definitions of safety criteria, range of appropriateness, noninferiority margin, and sample size. This pilot demonstrated an adequately powered definitive trial is feasible, which would provide evidence of DART safety and efficacy, ultimately informing potential for DART implementation. Trial Registration ClinicalTrials.gov NCT04904029; http://clinicaltrials.gov/ct2/show/NCT04904029 International Registered Report Identifier (IRRID) RR2-10.2196/31541

show abstract

Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial

Cited by 2 publications

References 54 publications

Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: Iterative, Convergent Mixed Methods Study

Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: Iterative, Convergent Mixed Methods Study

Evaluation of a Musculoskeletal Digital Assessment Routing Tool (DART): Crossover Noninferiority Randomized Pilot Trial

Contact Info

Product

Resources

About