Comparison of the validity, perceived usefulness and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability: a study protocol

Abstract

IntroductionResearch has shown that improvements to the usability of medication alert systems are needed. For designers and decisions-makers to assess usability of their alert systems, two paper-based tools are currently available: the instrument for evaluating human-factors principles in medication-related decision support alerts (I-MeDeSA) and the tool for evaluating medication alerting systems (TEMAS). This study aims to compare the validity, usability and usefulness of both tools to identify their strengths and limitations and assist designers and decision-makers in making an informed decision about which tool is most suitable for assessing their current or prospective system.Methods and analysisFirst, TEMAS and I-MeDeSA will be translated into French. This translation will be validated by three experts in human factors. Then, in 12 French hospitals with a medication alert system in place, staff with expertise in the system will evaluate their alert system using the two tools successively. After the use of each tool, participants will be asked to fill in the System Usability Scale (SUS) and complete a survey on the understandability and perceived usefulness of each tool. Following the completion of both assessments, participants will be asked to nominate their preferred tool and relay their opinions on the tools. The design philosophy of TEMAS and I-MeDeSA differs on the calculation of a score, impacting the way the comparison between the tools can be performed. Convergent validity will be evaluated by matching the items of the two tools with respect to the usability dimensions they assess. SUS scores and answers to the survey will be statistically compared for I-MeDeSA and TEMAS to identify differences. Free-text responses in surveys will be analysed using an inductive approach.Ethics and disseminationEthical approval is not required in France for a study of this nature. The results will be published in a peer-reviewed journal.

Qualitative study of challenges with recruitment of hospitals into a cluster controlled trial of clinical decision support in Australia

“…This study highlighted that this over-reliance is particularly a problem for controlled trials of CDS if a specific intervention is a frequently used form of CDS, like DDI alerts, or if the user has transferred from a different organisation, which is not uncommon. Ensuring end-users are aware of available CDS within an EMR, via good alert design (eg, visibility of available alerts in an EMR to end-users)31 and training, is critical for minimising inappropriate over-reliance on CDS.…”

Comparison of the validity, perceived usefulness, and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability

“…The method in our planned study protocol [20] was modified slightly due to variations in the context of implementation. The study included two main components: (1) assessment of the convergent validity of both tools, and (2) assessment of the perceived understandability, usability and usefulness of each tool and of users' preferences.…”

Cognitive task analysis of clinicians’ drug–drug interaction management during patient care and implications for alert design

“…Recent studies focus on DDI alert usability. Useful recommendations9 10 and alert evaluation tools,11 12 such as I-MedDeSA13 and TEMAS,14 have been published based on human factors engineering design principles. Newer DDI alert designs include optional Internet links, such as to detailed DDI monographs, for decision-making 10 15.…”