Meteorologists’ Interpretations of Storm-Scale Ensemble-Based Forecast Guidance

Abstract

During the 2017 Spring Forecasting Experiment in NOAA’s Hazardous Weather Testbed, 62 meteorologists completed a survey designed to test their understanding of forecast uncertainty. Survey questions were based on probabilistic forecast guidance provided by the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e). A mix of 20 multiple-choice and open-ended questions required participants to explain basic probability and percentile concepts, extract information using graphical representations of uncertainty, and determine what type of weather scenario the graphics depicted. Multiple-choice questions were analyzed using frequency counts, and open-ended questions were analyzed using thematic coding methods. Of the 18 questions that could be scored, 60%–96% of the participants’ responses aligned with the researchers’ intended response. Some of the most challenging questions proved to be those requiring qualitative explanations, such as to explain what the 70th-percentile value of accumulated rainfall represents in an ensemble-based probabilistic forecast. Additionally, participants providing answers not aligning with the intended response oftentimes appeared to consider the given information with a deterministic rather than probabilistic mindset. Applications of a deterministic mindset resulted in tendencies to focus on the worst-case scenario and to modify understanding of probabilistic concepts when presented with different variables. The findings from this survey support the need for improved basic and applied training for the development, interpretation, and use of probabilistic ensemble forecast guidance. Future work should collect data for a larger sample size to examine the knowledge gaps across specific user groups and to guide development of probabilistic forecast training tools.

Visualizing Uncertainty for Non-Expert End Users: The Challenge of the Deterministic Construal Error

“…For example, one study investigated professional weather forecasters understanding of a chart color-coded in shades of red and blue, indicating probability of accumulated rainfall greater than 0.01 inch. A full third of forecasters interpreted the increased color intensity (red in this case) as greater accumulation rather than greater likelihood (Wilson et al, 2019). Another example is a study of colorcoded climate outlook graphics for temperature (Gerst et al, 2020), intended for use by the U.S. National Oceanic and Atmospheric Administration (NOAA).…”

Evaluation of an experimental Warn‐on‐Forecast 3DVAR analysis and forecast system on quasi‐real‐time short‐term forecasts of high‐impact weather events

“…For example, extreme or hazardous weather events such as tornadoes, hail, and damaging winds usually demand quick delivery of forecast products. Despite emerging computer technologies, running a large number of high‐resolution ensemble members remains prohibitively expensive and time‐consuming; (c) To decide what uncertainty information (e.g., probabilities, forecaster confidence) is actually needed and how to communicate it to the public is a challenging task (Wilson et al ., 2019; Demuth et al ., 2020). Nevertheless, a poor decision might be made if based only on a single‐value forecast without accounting for any uncertainty information.…”

Test of a Weather-Adaptive Dual-Resolution Hybrid Warn-on-Forecast Analysis and Forecast System for Several Severe Weather Events

“…The system uses the WRF-ARW as its dynamic core and includes two components, the WRF-DART ensemble analysis and forecast component (NEWS-e) and a deterministic hybrid 3DEnVAR analysis and forecast component (NEWS-var). The NEWS-e component has been extensively tested in HWT spring experiments in recent years (e.g., Jones et al 2018;Skinner et al 2018;Wilson et al 2019). The NEWS-var component was tested as a high-resolution analysis system for severe weather (e.g., Gao et al 2013;Calhoun et al 2014), but has not been extensively tested as a short-term forecast system.…”