In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation

Abstract

Understanding and accounting for uncertainty is critical to effectively reasoning about visualized data. However, evaluating the impact of an uncertainty visualization is complex due to the difficulties that people have interpreting uncertainty and the challenge of defining correct behavior with uncertainty information. Currently, evaluators of uncertainty visualization must rely on general purpose visualization evaluation frameworks which can be ill-equipped to provide guidance with the unique difficulties of assessing judgments under uncertainty. To help evaluators navigate these complexities, we present a taxonomy for characterizing decisions made in designing an evaluation of an uncertainty visualization. Our taxonomy differentiates six levels of decisions that comprise an uncertainty visualization evaluation: the behavioral targets of the study, expected effects from an uncertainty visualization, evaluation goals, measures, elicitation techniques, and analysis approaches. Applying our taxonomy to 86 user studies of uncertainty visualizations, we find that existing evaluation practice, particularly in visualization research, focuses on Performance and Satisfaction-based measures that assume more predictable and statistically-driven judgment behavior than is suggested by research on human judgment and decision making. We reflect on common themes in evaluation practice concerning the interpretation and semantics of uncertainty, the use of confidence reporting, and a bias toward evaluating performance as accuracy rather than decision quality. We conclude with a concrete set of recommendations for evaluators designed to reduce the mismatch between the conceptualization of uncertainty in visualization versus other fields.

Effects of Uncertainty Visualization on Map-Based Decision Making Under Time Pressure

“…Overall, results are mixed concerning the effects of uncertainty visualizations: uncertainty visualizations can result in more accurate decisions (Andre and Cutler, 1998;Greis et al, 2018), but accuracy can also remain unchanged compared to decisions made without uncertainty (Leitner and Buttenfield, 2000;Riveiro et al, 2014). Results are likely mixed since visualization choice has a large effect on comprehension, as specific encodings of uncertainty visualizations have been shown to be more effective than others at facilitating accurate judgments and reducing misconceptions (Cheong et al, 2016;Ruginski et al, 2016;Hullman et al, 2018). Others have argued that individuals take specific decision strategies when presented with uncertainty, which may or may not change accuracy.…”

“…H1a: Uncertainty visualizations will influence map-based decisions regardless of time pressure. This hypothesis is based on the previously mentioned research showing that uncertainty visualizations have an influence on map-based decisions without time pressure (e.g., Wilkening and Fabrikant, 2011;Hullman et al, 2018). In order to test this hypothesis, the results of complex map-based decisions are assessed both objectively (accuracy, response time) and subjectively (decision confidence).…”

“…Similarly, our study was limited to a cartographic display in order to simulate a stressful decision-making scenario with geographic data. However, work shows that contextual variables, such as narrative framings and data domain, can affect visualization interpretation (Hullman and Diakopoulos, 2011;Correll and Gleicher, 2014;Hullman et al, 2018). These contexts have also been shown to affect behavior and decisionmaking under conditions of risk and uncertainty (Kübler et al, 2019).…”

Exploring the Effects of Aggregation Choices on Untrained Visualization Users' Generalizations From Data

“…As inferential statements, generalizations provide information about how visualization users think about and place confidence in data. We can assess generalizations from an accuracy perspective, as commonly used in visualization evaluations [LBI*12, IIS*14, HQC*19], asking whether different visual aggregation strategies lead to systematic differences in generalization accuracy. Recent work by Zgraggen et al .…”

Visual Analysis of the Exact Solution of a Heat Equation

“…Good visualization of the solution is an important issue, as visualization provides quick but approximate insights to the solution [1][2][3][4][5]. Quality of visualization of the solution depends on many factors such as the display device, human eye factors, sampling rate, interpolation technique, and bit depth of the image.…”