[1] In this paper we continue the community-wide rigorous modern space weather model validation efforts carried out within GEM, CEDAR and SHINE programs. In this particular effort, in coordination among the Community Coordinated Modeling Center (CCMC), NOAA Space Weather Prediction Center (SWPC), modelers, and science community, we focus on studying the models' capability to reproduce observed ground magnetic field fluctuations, which are closely related to geomagnetically induced current phenomenon. One of the primary motivations of the work is to support NOAA SWPC in their selection of the next numerical model that will be transitioned into operations. Six geomagnetic events and 12 geomagnetic observatories were selected for validation. While modeled and observed magnetic field time series are available for all 12 stations, the primary metrics analysis is based on six stations that were selected to represent the high-latitude and mid-latitude locations. Events-based analysis and the corresponding contingency tables were built for each event and each station. The elements in the contingency table were then used to calculate Probability of Detection (POD), Probability of False Detection (POFD) and Heidke Skill Score (HSS) for rigorous quantification of the models' performance. In this paper the summary results of the metrics analyses are reported in terms of POD, POFD and HSS. More detailed analyses can be carried out using the event by event contingency tables provided as an online appendix. An online interface built at CCMC and described in the supporting information is also available for more detailed time series analyses.

Ensemble modeling of coronal mass ejections (CMEs) provides a probabilistic forecast of CME arrival time which includes an estimation of arrival time uncertainty from the spread and distribution of predictions and forecast confidence in the likelihood of CME arrival. The real-time ensemble modeling of CME propagation uses the Wang-Sheeley-Arge (WSA)-ENLIL+Cone model installed at the Community Coordinated Modeling Center (CCMC) and executed in real-time at the CCMC/Space Weather Research Center. The current implementation of this ensemble modeling method evaluates the sensitivity of WSA-ENLIL+Cone model simulations of CME propagation to initial CME parameters. We discuss the results of real-time ensemble simulations for a total of 35 CME events which occurred between January 2013 -July 2014. For the 17 events where the CME was predicted to arrive at Earth, the mean absolute arrival time prediction error was 12.3 hours, which is comparable to the errors reported in other studies. For predictions of CME arrival at Earth the correct rejection rate is 62%, the false-alarm rate is 38%, the correct alarm ratio is 77%, and false alarm ratio is 23%. The arrival time was within the range of the ensemble arrival predictions for 8 out of 17 events. The Brier Score for CME arrival predictions is 0.15 (where a score of 0 on a range of 0 to 1 is a perfect forecast), which indicates that on average, the predicted probability, or likelihood, of CME arrival is fairly accurate. The reliability of ensemble CME arrival predictions is heavily dependent on the initial distribution of CME input parameters (e.g. speed, direction, and width), particularly the median and spread. Preliminary analysis of the probabilistic forecasts suggests undervariability, indicating that these ensembles do not sample a wide enough spread in CME input parameters. Prediction errors can also arise from ambient model parameters, the accuracy of the solar wind background derived from coronal maps, or other model limitations. Finally, predictions of the K P geomagnetic index differ from observed values by less than one for 11 out of 17 of the ensembles and K P prediction errors computed from the mean predicted K P show a mean absolute error of 1.3.

Modeling is an important tool in understanding physical processes in the space weather. Model performance studies evaluate the quality of model operation by comparing its output to a measurable parameter of interest. In this paper we studied the performance of the combination of the halo coronal mass ejection (CME) analytical cone model and ENLIL three‐dimensional MHD heliosphere model. We examined the CME arrival time and magnitude of impact at 1 AU for different geoeffective events, including the October 2003 Halloween Storm and the 14 December 2006 storm CMEs. The results of the simulation are compared with the ACE satellite observations. The comparison of the simulation results with the observations demonstrates that ENLIL cone model performs better compared to reference mean velocity and empirical models.

1] Acquiring quantitative metrics-based knowledge about the performance of various space physics modeling approaches is central for the space weather community. Quantification of the performance helps the users of the modeling products to better understand the capabilities of the models and to choose the approach that best suits their specific needs. Further, metrics-based analyses are important for addressing the differences between various modeling approaches and for measuring and guiding the progress in the field. In this paper, the metrics-based results of the ground magnetic field perturbation part of the Geospace Environment Modeling 2008-2009 Challenge are reported. Predictions made by 14 different models, including an ensemble model, are compared to geomagnetic observatory recordings from 12 different northern hemispheric locations. Five different metrics are used to quantify the model performances for four storm events. It is shown that the ranking of the models is strongly dependent on the type of metric used to evaluate the model performance. None of the models rank near or at the top systematically for all used metrics. Consequently, one cannot pick the absolute "winner": the choice for the best model depends on the characteristics of the signal one is interested in. Model performances vary also from event to event. This is particularly clear for root-mean-square difference and utility metric-based analyses. Further, analyses indicate that for some of the models, increasing the global magnetohydrodynamic model spatial resolution and the inclusion of the ring current dynamics improve the models' capability to generate more realistic ground magnetic field fluctuations.