Examining the Impact of Training and Feedback on Visualization-Supported Decision Making under Uncertainty

Abstract

Empirical evaluations of uncertainty visualizations often employ complex experimental tasks to ensure ecological validity. However, if training for such tasks is not sufficient for naïve participants, differences in performance could be due to the visualizations or to differences in task comprehension, making interpretation of findings problematic. Research has begun to assess how training is related to performance on decision-making tasks using uncertainty visualizations. This study continues this line of research by investigating how training, in general, and feedback, in particular, affect performance on a simulated resource allocation task. Additionally, we examined how this alters metacognition and workload to produce differences in cognitive efficiency. Our results suggest that, on a complex decision-making task, training plays a critical role in performance with respect to accuracy, subjective workload, and cognitive efficiency. This study has implications for improving research on complex decision making, and for designing more efficacious training interventions to assess uncertainty visualizations.

Examining Training Comprehension and External Cognition in Evaluations of Uncertainty Visualizations to Support Decision Making

“…The present study has two overarching goals. First, this study aims to replicate recent findings emphasizing the importance of training in uncertainty visualization research (e.g., Fiore et al, 2019;Song et al, 2018). In doing so, we contribute to this growing area of research on how to improve the design of empirical evaluations of uncertainty visualizations.…”

“…This highlights the importance of understanding how varying forms of training may influence task comprehension and potentially distort the apparent effectiveness of uncertainty visualizations. However, a related study found that regardless of variations in training administration, participants who demonstrated relatively high training comprehension outperformed those who demonstrated relatively low training comprehension (Song et al, 2018). In addition to considering differences in decision making and task familiarity between experts and non-experts, we also propose that equal consideration should be given when it comes to differences within non-expert participant samples.…”

“…Despite the amount of interest in understanding how to design effective visualizations of uncertainty, a potential limitation of empirical evaluations in this area concerns the inherent difficulty non-expert participants face when they engage in tasks requiring domain-specific knowledge. For instance, prior work examining this limitation has shown that performance on a decision-making task utilizing uncertainty visualizations may vary not just due to the effectiveness of the visualization, but also as a function of how well participants comprehend the task (Fiore et al, 2019;Song et al, 2018).…”

Examining Enhanced Learning Diagnostics in Virtual Reality Flight Trainers

“…When considering VR simulators for testing, they may provide an additional diagnostic capable of better predicting training efficacy. For example, previous research shows that different types of knowledge acquisition assessments predict transfer performance on complex decision-making tasks (Fiore et al, 2002;Song et al, 2018). Recently, Song and colleagues (2019) found that training comprehension was predictive of complex transfer-type tasks assessing decision making under uncertainty.…”