In many human performance tasks, researchers assess performance by measuring both accuracy and response time. A number of theoretical and practical approaches have been proposed to obtain a single performance value that combines these measures, with varying degrees of success. In this report, we examine data from a common paradigm used in applied human factors assessment: a go/no-go vigilance task (Smith et al., 2019). We examined whether 12 different measures of performance were sensitive to the vigilance decrement induced by the design, and also examined how the different measures were correlated. Results suggest that most combined measures were slight improvements over accuracy or response time alone, with the most sensitive and representative result coming from the Linear Ballistic Accumulator model. Practical lessons for applying these measures are discussed.

The success of deep image classification networks has been met with enthusiasm and investment from both the academic community and industry. We hypothesize users will expect these systems to behave similarly to humans, and to succeed and fail in ways humans do. To investigate this, we tested six popular image classifiers on imagery from ten tool categories, examining how 17 visual transforms impacted both human and AI classification. Results showed that (1) none of the visual transforms we examined produced substantial impairment for human recognition; (2) human errors were limited to mostly to functional confusions; (3) almost all visual transforms impacted nearly every image classifier negatively and often catastrophically; (4) human expectations about performance of AI classifiers map more closely onto human error than AI performance; and (5) models trained with an enriched training set involving examples of the transformed imagery achieved improved performance but were not inoculated from error.