How Many Participants Are Required for Validation of Automated Vehicle Interfaces in User Studies?

Abstract

Empirical validation and verification procedures require the sophisticated development of research methodology. Therefore, researchers and practitioners in human–machine interaction and the automotive domain have developed standardized test protocols for user studies. These protocols are used to evaluate human–machine interfaces (HMI) for driver distraction or automated driving. A system or HMI is validated in regard to certain criteria that it can either pass or fail. One important aspect is the number of participants to include in the study and the respective number of potential failures concerning the pass/fail criteria of the test protocol. By applying binomial tests, the present work provides recommendations on how many participants should be included in a user study. It sheds light on the degree to which inferences from a sample with specific pass/fail ratios to a population is permitted. The calculations take into account different sample sizes and different numbers of observations within a sample that fail the criterion of interest. The analyses show that required sample sizes increase to high numbers with a rising degree of controllability that is assumed for a population. The required sample sizes for a specific controllability verification (e.g., 85%) also increase if there are observed cases of fails in regard to the safety criteria. In conclusion, the present work outlines potential sample sizes and valid inferences about populations and the number of observed failures in a user study.

Test procedure for the evaluation of partially automated driving HMI including driver monitoring systems in driving simulation

“…From these rationales the recommendation for the sample size is derived: For the evaluation of the fulfilment of the requirements it is recommended to examine at least 20 subjects plus additional over-recruitment, so that finally at least 20 valid data set per HMI or DMS condition are available (one of the prerequisites for a data set to be considered valid is that a subject understood the instruction). For a more detailed overview on sample sizes in validation and verification studies and potential number of participants that may fail a criterion (but where the protocol still allows a signoff) see Forster et al., [8] .…”