Sensitivity of Skill Score Metric to Validate Lagrangian Simulations in Coastal Areas: Recommendations for Search and Rescue Applications

Abstract

Search and rescue (SAR) modeling applications, mostly based on Lagrangian tracking particle algorithms, rely on the accuracy of met-ocean forecast models. Skill assessment methods are therefore required to evaluate the performance of ocean models in predicting particle trajectories. The Skill Score (SS), based on the Normalized Cumulative Lagrangian Separation (NCLS) distance between simulated and satellite-tracked drifter trajectories, is a commonly used metric. However, its applicability in coastal areas, where most of the SAR incidents occur, is difficult and sometimes unfeasible, because of the high variability that characterizes the coastal dynamics and the lack of drifter observations. In this study, we assess the performance of four models available in the Ibiza Channel (Western Mediterranean Sea) and evaluate the applicability of the SS in such coastal risk-prone regions seeking for a functional implementation in the context of SAR operations. We analyze the SS sensitivity to different forecast horizons and examine the best way to quantify the average model performance, to avoid biased conclusions. Our results show that the SS increases with forecast time in most cases. At short forecast times (i.e., 6 h), the SS exhibits a much higher variability due to the short trajectory lengths observed compared to the separation distance obtained at timescales not properly resolved by the models. However, longer forecast times lead to the overestimation of the SS due to the high variability of the surface currents. Findings also show that the averaged SS, as originally defined, can be misleading because of the imposition of a lower limit value of zero. To properly evaluate the averaged skill of the models, a revision of its definition, the so-called SS∗, is recommended. Furthermore, whereas drifters only provide assessment along their drifting paths, we show that trajectories derived from high-frequency radar (HFR) effectively provide information about the spatial distribution of the model performance inside the HFR coverage. HFR-derived trajectories could therefore be used for complementing drifter observations. The SS is, on average, more favorable to coarser-resolution models because of the double-penalty error, whereas higher-resolution models show both very low and very high performance during the experiments.

Evaluating High-Frequency radar data assimilation impact in coastal ocean operational modelling

“…All experiments present a larger SS outside the HFR coverage area. This in mainly due to the characteristics of the trajectories north of the island of Ibiza, where the circulation is dominated by the Balearic current, with a more steady northeastward flow, generally reproduced in the model, as previously described by (Révelard et al, 2021). .…”

“…Three different kinds of surface drifters (ODi, MDOi and CODEi) were employed, all drifting at a depth between the surface and 1m. No significant wind drag is expected for these drifters (more details can be found in Révelard et al (2021) or Barth et al (2021)). Virtual particle drifts were then computed using model surface currents.…”

Evaluating high-frequency radar data assimilation impact in coastal ocean operational modelling

“…1) with a horizontal resolution around 2 km and 32 vertical sigma levels (resulting in a vertical resolution between 1 and 2 m at the surface). The WMOP system is used to produce daily forecasts of the regional ocean circulation, which is used for a wide range of applications including search and rescue and analysis of plastic, parasite, or larval dispersion (Calò et al, 2018;Ruiz-Orejón et al, 2019;Cabanellas-Reboredo et al, 2019;Compa et al, 2020;Torrado et al, 2021;Kersting et al, 2020;Révelard et al, 2021).…”

“…As previously stated, 13 surface drifters were deployed in the HFR coverage area during the simulated period, as described in Lana et al (2016). Three different kinds of surface drifters (ODi, MDOi, and CODEi) were employed, all drifting at a depth between the surface and 1 m. No significant wind drag is expected for these drifters (more details can be found in Révelard et al, 2021or Barth et al, 2021a. Virtual particle drifts were then computed using model surface currents.…”

“…More recently, hourly reconstructed total currents have also been employed using both sequential (Ren et al, 2016;Paduan and Shulman, 2004) and variational data assimilation schemes such as 4D-Var (Zhang et al, 2010;Wilkin and Hunter, 2013;Yu et al, 2016). However, depending on the model setup and the oceanic processes of interest, the use of hourly data may not be the most appropriate, as for instance in Kerry et al (2016), where radial speeds and angles are spatially averaged onto the model grid and a 24 h boxcaraveraging filter is used to remove tides and inertial oscillations that are not resolved by the model.…”