Testing the Reliability of Interpretable Neural Networks in Geoscience Using the Madden-Julian Oscillation

Abstract

Abstract. We test the reliability of two neural network interpretation techniques, backward optimization and layerwise relevance propagation, within geoscientific applications by applying them to a commonly studied geophysical phenomenon, the Madden-Julian Oscillation. The Madden-Julian Oscillation is a multi-scale pattern within the tropical atmosphere that has been extensively studied over the past decades, which makes it an ideal test case to ensure the interpretability methods can recover the current state of knowledge regarding its spatial structure. The neural networks can, indeed, reproduce the current state of knowledge and can also provide new insights into the seasonality of the Madden-Julian Oscillation and its relationships with atmospheric state variables. The neural network identifies the phase of the Madden-Julian Oscillation twice as accurately as linear regression, which means that nonlinearities used by the neural network are important to the structure of the Madden-Julian Oscillation. Interpretations of the neural network show that it accurately captures the spatial structures of the Madden-Julian Oscillation, suggest that the nonlinearities of the Madden-Julian Oscillation are manifested through the uniqueness of each event, and offer physically meaningful insights into its relationship with atmospheric state variables. We also use the interpretations to identify the seasonality of the MJO, and find that the conventionally defined extended seasons should be shifted later by one month. More generally, this study suggests that neural networks can be reliably interpreted for geoscientific applications and may thereby serve as a dependable method for testing geoscientific hypotheses.

Using Simple, Explainable Neural Networks to Predict the Madden‐Julian Oscillation

“…At lead 0, where the classification model is identifying the MJO, the phase of active MJO events are correctly predicted with an accuracy of ∼80% (Figure 6), comparable to Toms et al. (2019), despite differences in our input variables, data pre‐processing, MJO index, and ANN complexity. The majority of incorrectly predicted active MJO events at short leads are near the boundary between two RMM phases and predictions are often incorrect by only one phase.…”

Physics-informed machine learning: case studies for weather and climate modelling

“…Gagne et al [81] used feature importance and feature optimization to interpret their CNN model for predicting the probability of severe hailstorms and found that the model synthesized information about the environment and storm morphology that is consistent with our current understanding of the physics of hailstorms. Toms et al [82] developed interpretable NNs for the geosciences and showed their usefulness and reliability in improving our understanding of the Madden-Julian oscillation [83]. Brenowitz et al [43] developed an interpretability framework specialized for analysis of the relationship between offline skill versus online coupled prognostic performance for ML parameterizations of convection.…”

Assessing Decadal Predictability in an Earth‐System Model Using Explainable Neural Networks

“…Our usage of LRP, therefore, offers insights into which patterns of SST variability lend predictability of decadal surface temperature anomalies over continental North America within CESM2. A more detailed discussion of LRP and its applicability to Earth-system research is discussed in Toms, Barnes, and Ebert-Uphoff (2020), and additional applications are available in Barnes et al (2020), Ebert-Uphoff and Hilburn (2020), and Toms, Kashinath, et al (2020).…”

“…The specific usage of neural network interpretation techniques ranges substantially across such studies, however, the interpretations can be used as either direct or indirect tools for scientific discovery. For example, interpretation efforts can be either a secondary objective by ensuring a network's reasoning is consistent with existing physical theory (e.g., Brenowitz et al, 2020;Ebert-Uphoff and Hilburn, 2020;Toms, Kashinath, et al, 2020), or the primary objective, with their usage focused on discovering new patterns of Earth-system variability (e.g., Barnes et al, 2020;. Here, we focus on the latter application, whereby we use neural networks to identify predictable modes of Earth-system variability on decadal timescales in a fully coupled Earth-system model.An extensive body of literature exists on theoretical and observed sources of decadal predictability, and more recently, on the development of operational decadal prediction systems (Yeager et al, 2018).…”