Visual Reasoning Strategies for Effect Size Judgments and Decisions

Kale, Alex; Kay, Matthew; Hullman, Jessica

doi:10.1109/tvcg.2020.3030335

Cited by 87 publications

(128 citation statements)

References 46 publications

Supporting

Mentioning

123

Contrasting

Order By: Relevance

“…A growing body of research finds that visualizations that show distributional information in frequency framing can improve accuracy and memory compared to visualizations that show only probability distributions and/or summary statistics (e.g., Kay et al, 2016;Ruginski et al, 2016;Hullman et al, 2017;Padilla et al, 2017;Fernandes et al, 2018;Kale et al, 2020; see examples of distributional visualizations and those that use frequency framing in Figure 1). A reliable finding across uncertainty visualization research is that static interval plots, such as the ubiquitous 95% confidence interval, can lead to errors and biases (e.g., Belia et al, 2005;Joslyn and LeClerc, 2012;Padilla et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

“…A reliable finding across uncertainty visualization research is that static interval plots, such as the ubiquitous 95% confidence interval, can lead to errors and biases (e.g., Belia et al, 2005;Joslyn and LeClerc, 2012;Padilla et al, 2017). Many studies find that increasing the expressiveness of an interval plot by displaying distributional information can improve performance, for example, with quantile dot plots (Kay et al, 2016;Fernandes et al, 2018;Kale et al, 2020), hypothetical outcome plots (Hullman et al, 2015;Kale et al, 2018), ensemble plots (Ruginski et al, 2016;Padilla et al, 2017), gradient plots, and violin plots (Correll and Gleicher, 2014). Among the visualizations that show distributional information, those that include frequency framing, specifically quantile dot plots and hypothetical outcome plots, have been found to outperform other distributional visualizations (Hullman et al, 2015;Kay et al, 2016;Fernandes et al, 2018;Kale et al, 2018Kale et al, , 2020.…”

Section: Introductionmentioning

confidence: 99%

“…Many studies find that increasing the expressiveness of an interval plot by displaying distributional information can improve performance, for example, with quantile dot plots (Kay et al, 2016;Fernandes et al, 2018;Kale et al, 2020), hypothetical outcome plots (Hullman et al, 2015;Kale et al, 2018), ensemble plots (Ruginski et al, 2016;Padilla et al, 2017), gradient plots, and violin plots (Correll and Gleicher, 2014). Among the visualizations that show distributional information, those that include frequency framing, specifically quantile dot plots and hypothetical outcome plots, have been found to outperform other distributional visualizations (Hullman et al, 2015;Kay et al, 2016;Fernandes et al, 2018;Kale et al, 2018Kale et al, , 2020. Best practices in uncertainty visualization suggest distributional information be displayed using frequency framing when possible and in a way that does not allow the viewer to fixate on summary information such as a mean (Kale et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

“…Among the visualizations that show distributional information, those that include frequency framing, specifically quantile dot plots and hypothetical outcome plots, have been found to outperform other distributional visualizations (Hullman et al, 2015;Kay et al, 2016;Fernandes et al, 2018;Kale et al, 2018Kale et al, , 2020. Best practices in uncertainty visualization suggest distributional information be displayed using frequency framing when possible and in a way that does not allow the viewer to fixate on summary information such as a mean (Kale et al, 2020). For a full review of modern uncertainty visualization techniques and theory, see Padilla et al (2020).…”

Section: Introductionmentioning

confidence: 99%

“…The viewer can count the number of dots in a given range to determine the probability of the nightly low temperature falling within that range. In several empirical studies, researchers have found that quantile dotplots improve memory of distributional information and lead to more consistent probability estimates compared to probability density plots (Kay et al, 2016;Hullman et al, 2017;Kale et al, 2020). More studies have found that quantile dotplots outperform interval plots, density plots, and textural descriptions of uncertainty for decisions with risk (Fernandes et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Uncertain About Uncertainty: How Qualitative Expressions of Forecaster Confidence Impact Decision-Making With Uncertainty Visualizations

et al. 2021

Self Cite

View full text Add to dashboard Cite

When forecasting events, multiple types of uncertainty are often inherently present in the modeling process. Various uncertainty typologies exist, and each type of uncertainty has different implications a scientist might want to convey. In this work, we focus on one type of distinction between direct quantitative uncertainty and indirect qualitative uncertainty. Direct quantitative uncertainty describes uncertainty about facts, numbers, and hypotheses that can be communicated in absolute quantitative forms such as probability distributions or confidence intervals. Indirect qualitative uncertainty describes the quality of knowledge concerning how effectively facts, numbers, or hypotheses represent reality, such as evidence confidence scales proposed by the Intergovernmental Panel on Climate Change. A large body of research demonstrates that both experts and novices have difficulty reasoning with quantitative uncertainty, and visualizations of uncertainty can help with such traditionally challenging concepts. However, the question of if, and how, people may reason with multiple types of uncertainty associated with a forecast remains largely unexplored. In this series of studies, we seek to understand if individuals can integrate indirect uncertainty about how “good” a model is (operationalized as a qualitative expression of forecaster confidence) with quantified uncertainty in a prediction (operationalized as a quantile dotplot visualization of a predicted distribution). Our first study results suggest that participants utilize both direct quantitative uncertainty and indirect qualitative uncertainty when conveyed as quantile dotplots and forecaster confidence. In manipulations where forecasters were less sure about their prediction, participants made more conservative judgments. In our second study, we varied the amount of quantified uncertainty (in the form of the SD of the visualized distributions) to examine how participants’ decisions changed under different combinations of quantified uncertainty (variance) and qualitative uncertainty (low, medium, and high forecaster confidence). The second study results suggest that participants updated their judgments in the direction predicted by both qualitative confidence information (e.g., becoming more conservative when the forecaster confidence is low) and quantitative uncertainty (e.g., becoming more conservative when the variance is increased). Based on the findings from both experiments, we recommend that forecasters present qualitative expressions of model confidence whenever possible alongside quantified uncertainty.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Uncertain About Uncertainty: How Qualitative Expressions of Forecaster Confidence Impact Decision-Making With Uncertainty Visualizations

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

Visualizing Uncertainty in Time Series Forecasts: The Impact of Uncertainty Visualization on Users' Confidence, Algorithmic Advice Utilization, and Forecasting Performance

Leffrang,

Müller

2024

Journal of Forecasting

View full text Add to dashboard Cite

Time series forecasts are always associated with uncertainty. However, experimental studies on the impact of uncertainty communication provide inconclusive results on the effect of providing this uncertainty to end users. In this study, we examine the impact of uncertainty visualizations on advice utilization in the context of time series forecasts with line charts. Based on a literature review, we identified probabilistic framing versus frequency framing as a theoretical foundation for studying the topic. We then used the Judge Advisor System (JAS) as a framework to create an experimental design with two treatments (95% prediction interval [PI] and ensemble plots), one control group (point plot), and various mediating variables (e.g., confidence, graph literacy, and domain knowledge). The results of an online experiment () indicate a U‐shaped relation between uncertainty visualization and forecasting performance. Additionally, we examine how confidence, advice utilization, and other factors mediate the effect of uncertainty visualizations. This paper highlights the benefits of PI plots for researchers and practitioners engaged in the development of effective uncertainty visualizations for decision‐making processes.

show abstract

Improving Evaluation Using Visualization Decision-Making Models: A Practical Guide

Bancilhon,

Padilla,

Ottley

2023

Visualization Psychology

View full text Add to dashboard Cite

Visual Reasoning Strategies for Effect Size Judgments and Decisions

Cited by 87 publications

References 46 publications

Uncertain About Uncertainty: How Qualitative Expressions of Forecaster Confidence Impact Decision-Making With Uncertainty Visualizations

Uncertain About Uncertainty: How Qualitative Expressions of Forecaster Confidence Impact Decision-Making With Uncertainty Visualizations

Visualizing Uncertainty in Time Series Forecasts: The Impact of Uncertainty Visualization on Users' Confidence, Algorithmic Advice Utilization, and Forecasting Performance

Improving Evaluation Using Visualization Decision-Making Models: A Practical Guide

Contact Info

Product

Resources

About