How many testers are needed to assure the usability of medical devices?

Abstract

A note on versions:The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the repository url above for details on accessing the published version and note that access may require a subscription.  The assessment of usability and its place in the management of product safety is an increasingly important aspect of medical device development. Medical device manufacturers, especially in small companies, do not have sufficient expertise or knowledge about usability and human factors and it is hard for them suitably to address all of the steps required by relevant standards. Despite a rigorous framework for designing and assessing a product, an increasing number of medical devices are recalled each year because of safety concerns. There is substantial evidence that testing a device with mandated sample sizes could lead manufacturers to evaluate a product without having real control over the reliability of the assessment. A new approach to usability data management applied in this paper, called the 'Grounded Procedure', drives manufacturers to estimate the sample size needed to identify a given proportion of interaction problems and to inform critical product decisions. Using the Grounded Procedure could enable manufacturers to increase the usability and the safety of their medical devices and help practitioners to check the representativeness of the evaluation cohorts, to analyse significance of specific usability problems, and to rethink the user selection criteria for validation testing. Summary 3Before releasing a product, manufacturers have to follow a regulatory framework and meet standards, producing reliable evidence that the device presents low levels of risk in use. There is, though, a gap between the needs of the manufacturers to conduct usability testing whilst managing their costs, and the requirements of authorities for representative evaluation data. A key issue here is the number of users that should complete this evaluation to provide confidence in a product's safety. This paper reviews the FDA's indication that a sample composed of 15 participants per major group (or a minimum of 25 users) should be enough to identify 90-97% of the usability problems and argues that a more nuanced approach to determining sample size (which would also fit well with the FDA's own concerns expressed in [1]) would be beneficial. The paper will show that there is no a priori cohort size that can guarantee a reliable assessment, a point stressed by the FDA in the appendices to its guidance, but that manufacturers can terminate the assessment when appropriate by using a specific approach -illustrated in this paper through a case study -called the 'Grounded Procedure'.4

Using Patient Feedback to Optimize the Design of a Certolizumab Pegol Electromechanical Self-Injection Device: Insights from Human Factors Studies

“…HFS are used to observe patients’ interactions with devices and iteratively improve device design by listening to patient feedback. A review of medical device usability testing methods highlighted the value of HFS as part of a patient-centric design process, with user-feedback-led design more likely to result in patient satisfaction and correct use of the final device [ 11 ]. Furthermore, these studies form part of the USA FDA’s requirements to validate the usability and safety of medical devices [ 10 , 12 , 13 ], and help to demonstrate conformance with requirements of EU and international regulatory authorities, such as the European Parliament and Council of the European Communities’ medical devices directives and the International Electrotechnical Commission [ 13 – 15 ].…”

Development of an innovative double-chamber syringe for intravenous therapeutics and flushing: Nurses’ involvement through a human-centred approach

“…Prototypes play an important role in the development of new MDs allowing for the assessment of shape and form before manufacturing [49]. Thus, identifying and recruiting representative endusers is essential to ensure a reliable data set [50,51].…”

Portal Design for the Open Data Initiative: A Preliminary Study

“…While the number of evaluators is small, the purpose of this preliminary evaluation is to assess the adequacy of the current ODI data model before it is coded up as a functional portal system. It is envisaged that more formal evaluation of later iterations will involve a greater number of users to identify errors [10]. The technical review outcomes can be summarized under the broad headings of data entry considerations and dissemination considerations.…”