Comprehensive Assessment of Models and Events Using Library Tools (CAMEL) Framework: Time Series Comparisons

Abstract

The Comprehensive Assessment of Models and Events using Library Tools (CAMEL) framework leverages existing Community Coordinated Modeling Center services: Run‐on‐Request postprocessing tools that generate model time series outputs and the new Community Coordinated Modeling Center Metadata Registry that describes simulation runs using Space Physics Archive Search and Extract metadata. The new CAMEL visualization tool compares the modeled time series with observational data and computes a suite of skill scores such as Prediction Efficiency, Root‐Mean‐Square Error, and Symmetric Signed Percentage Bias. Model‐data pairs used for skill calculations are obtained considering a user‐selected maximum difference between the time of observation and the nearest model output. The system renders available data for all locations and time periods selected using interactive visualizations that allow the user to zoom, pan, and pick data values along traces. Skill scores are reported for each selected event or aggregated over all events for all participating model runs. Separately, scores are reported for all locations (satellites or stations) and for each location individually. We are building on past experiences with model‐data comparisons of magnetosphere and ionosphere model outputs from GEM2008, GEM‐CEDAR Electrodynamics Thermosphere Ionosphere, and the SWPC Operational Space Weather Model challenges. The CAMEL visualization tool is demonstrated using three validation studies: (a) Wang‐Sheeley‐Arge heliosphere simulations compared against OMNI solar wind data, (b) ground magnetic perturbations from several magnetosphere and ionosphere electrodynamics models as observed by magnetometers, and (c) electron fluxes from several ring current simulations compared to Radiation Belt Storm Probes Helium Oxygen Proton Electron instrument measurements, integrated over different energy ranges.

Unifying the Validation of Ambient Solar Wind Models

“…More details on SPASE metadata can be found in Fung et al (2022). As the backbone of the online platform, the Comprehensive Assessment of Models and Events Using Library Tools (CAMEL; Rastätter et al, 2019) framework hosted by NASA's CCMC will make the metadata accessible to the community online.…”

“…We use the existing Comprehensive Assessment of Models and Events Using Library Tools (CAMEL; Rastätter et al, 2019) framework to make our developments accessible to the community. The implementation of CAMEL consists of a front-end and a back-end reaching deep into CCMC's modeling infrastructure.…”

“…This way, model developers can make their models and updated versions, including a comprehensive validation analysis, readily accessible to the community. For more information on CAMEL, we refer the interested reader to Rastätter et al (2019).…”

Space Radiation and Plasma Effects on Satellites and Aviation: Quantities and Metrics for Tracking Performance of Space Weather Environment Models

“…All factors should be kept in mind for the fairness of the validation results. The CAMEL (Comprehensive Assessment of Models and Events based on Library tools) system to be discussed next (section ) will provide choices of different metrics (could be one or more) that are suitable for quantities/models at consideration (Rastätter et al, ). Besides metrics evaluation for individual events, evaluation of a model performance over multiple events or an extended time interval will also be carried out.…”

“…It is a framework to combine tools to perform model execution, postprocessing, and model evaluation. For details, please see the CAMEL paper of this special issue (Rastätter et al, ). This tool stores model outputs and observations for all validation studies, plots model and observations together, has built‐in variety of metrics, and is to incorporate features of the National Center for Atmospheric Research's Model Evaluation Tool (MET) through partnership with Tara Jensen, Barb Brown et al (Jensen & Brown, ).…”