Utility of Crowdsourced User Experiments for Measuring the Central Tendency of User Performance to Evaluate Error-Rate Models on GUIs

Abstract

The usage of crowdsourcing to recruit numerous participants has been recognized as beneficial in the human-computer interaction (HCI) field, such as for designing user interfaces and validating user performance models. In this work, we investigate its effectiveness for evaluating an error-rate prediction model in target pointing tasks. In contrast to models for operational times, a clicking error (i.e., missing a target) occurs by chance at a certain probability, e.g., 5%. Therefore, in traditional laboratory-based experiments, a lot of repetitions are needed to measure the central tendency of error rates. We hypothesize that recruiting many workers would enable us to keep the number of repetitions per worker much smaller. We collected data from 384 workers and found that existing models on operational time and error rate showed good fits (both R^2 > 0.95). A simulation where we changed the number of participants N_P and the number of repetitions N_repeat showed that the time prediction model was robust against small N_P and N_repeat, although the error-rate model fitness was considerably degraded. These findings empirically demonstrate a new utility of crowdsourced user experiments for collecting numerous participants, which should be of great use to HCI researchers for their evaluation studies.

Test-retest Reliability on Movement Times and Error Rates in Target Pointing

“…It is known that the error rate ER tends to be higher as 𝑊 decreases (e.g., [11,17,44]; see Figure 1a). However, this relationship between ER and 𝑊 is not linear; thus, predicting the ER requires a more probabilistic process.…”

“…Hereafter, italic lowercase letters 𝑎-𝑙 refer to empirical constants. This formulation has been used for predicting endpoint variability in mouse-based [44] and virtual-reality pointing tasks [48]. As shown by this equation, the distance from the initial cursor (or finger) position to the target does not affect the endpoint distribution if users have sufficient time to aim for the target [8,12,44,45].…”

“…This formulation has been used for predicting endpoint variability in mouse-based [44] and virtual-reality pointing tasks [48]. As shown by this equation, the distance from the initial cursor (or finger) position to the target does not affect the endpoint distribution if users have sufficient time to aim for the target [8,12,44,45]. We do not intend to run time-limited or ballistic (i.e., not using visual feedback) pointing tasks, where the target distance affects the distributions [5,18,26,32,41].…”

“…For a new target condition, we can accurately predict 𝜎 𝑥 by using Equation 1 then obtain the ER on the basis of the new 𝑊 and predicted 𝜎 𝑥 by using Equation 2. For example, Yamanaka conducted a mouse-pointing task with 𝑊 ranging from 8 to 78 pixels and obtained 𝑅 2 = 0.9966 for Equation 1 and 𝑅 2 = 0.9529 for Equation 2 in leave-one-out cross-validation, which demonstrated good prediction accuracy for unknown target conditions [44].…”

Utility of Crowdsourced User Experiments for Measuring the Central Tendency of User Performance: A Case of Error-Rate Model Evaluation in a Pointing Task

“…This article is an extended version of our previous work presented at the AAAI HCOMP 2021 conference (Yamanaka, 2021b).…”