Usability Testing of a Digitized Interventional Prehabilitation Tool for Health Care Professionals and Patients Before Major Surgeries: Formative and Summative Evaluation (Preprint)

Abstract

BACKGROUND The development of a medical device requires strict adherence to regulatory processes. Here the usability engineering process for The Prehab App is displayed. Prehabilitation in this context is a new area in surgery that trains, coaches, and advises patients in mental well-being, nutrition, and physical activity. As staff is permanently drained from clinical care, remote and digital solutions with real-time assessments of data including patient-related outcome reporting may simplify preparation before major surgeries. OBJECTIVE The hypothesis was, that formative and summative usability testing would achieve 80% interrater, and intra-rater reliability and consistency and that the safety-relevant scenarios would uncover undetected risks of the medical device (standalone software class IIa). METHODS Eight experts and eight patients were asked to evaluate paper-based mockups, followed by an evaluation of the minimal viable product of The Prehab App at least more than 8 weeks later after instruction and training. The experts had to perform 5 and the patients 6 usability scenarios. The evaluations were measured with the MARS (mobile app rating scale) and Trustworthiness checklists (range 0-64 with higher scores indicating trustworthiness), usability scenarios received evaluation with the ‘After scenario questionnaire’ and a judgment by an observer. The time of performing the scenarios was recorded. RESULTS MARS achieved constant scores of more than 4 out of 5 points for experts and patients. The Trustworthiness score was 51,3 ± 2,7 in the experts and 50,8 ± 2,1 in the lays group (p=0,68) in Task I. The interrater correlation, shown by the Fleiss-Kappa value was 0,87 [0,85;0,89] for all raters (n=18) reflecting almost perfect agreement, and 0,86 [0,82; 0,91] for the expert group (n=8) and 0,88 [0,84; 0,93] for the lays group (n=8) reflecting the almost perfect agreement between the raters. This indicated the high quality of the usability. The usability scenarios were performed with ease except for the onboarding part when connecting the wearable was required. This took a considerable amount of time and was recognized as a challenge for good usability. CONCLUSIONS The formative and summative evaluation of ‘The Prehab App’ design resulted in very good to acceptable results of the design and usability of the critical and safety-relevant areas of the medical device and standalone software. Usability testing improves medical devices early in the design and development process, reduces errors, mitigates risks, and delivers a profound ethical and medical justification for a randomized controlled trial of ‘The Prehab App’ in a remote setting as a next step in the development process. CLINICALTRIAL The trial received a positive vote by the Frankfurt University institutional review board (IRB) on November 11th, 2021 (2021-483-MDR) and was approved by the Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM) on February 7th, 2022 (German Medical Devices Information and Database System: DMIDS: 00013655; European Database on Medical Devices: EUDMAD: CIV-21-07-037311; German Study Registry: DRKS00026985).