Extremely randomized neural networks for constructing prediction intervals

Environmental Engel Curves: A Neural Network Approach

“…The concept of MC dropout is central to this novel literature on prediction intervals for NN models, see Srivastava et al (2014). An alternative to construct prediction intervals in a NN framework, recently proposed in Mancini et al (2021), explores the results for randomised trees and derives valid confidence intervals of the model predictions in finite samples. In this work, we focus on the latter two methodologies that are reviewed in the following subsections.…”

“…The EN network approach can be interpreted as an ensemble predictor for NN models. This methodology, formally introduced in Mancini et al (2021), is based on the extremely randomised trees approach proposed by Geurts et al (2006) for random forests.…”

“…Given the following ensemble, let $$$ {\overline{f}}_{\mathrm{EN}}\left({\mathbf{x}}_i\right) $$$ denote the ensemble predictor obtained from the EN network approach that is constructed as $$$$ {\overline{f}}_{\mathrm{EN}}\left({\mathbf{x}}_i\right)=\frac{1}{T}\sum \limits_{t=1}^T{f}_t\left({\mathbf{x}}_i;{\hat{\boldsymbol{\omega}}}^{(t)}\right),\kern0.3em \mathrm{for}\kern0.3em i=1,\dots, n. $$$$ where $$$ {\hat{\boldsymbol{\omega}}}^{(t)} $$$ denotes the parameter estimates obtained from fitting each sub‐network independently. Mancini et al (2021) show that the MSPE of the above ensemble predictor can be expressed as $$$$ \mathrm{MSPE}\left({\overline{f}}_{\mathrm{EN}}\left({\mathbf{x}}_i\right)\right)={\mu}_i^2+\frac{1}{T}{\sigma}_{\hat{\boldsymbol{\omega}}}^2\left({\mathbf{x}}_i\right)+\frac{T-1}{T}{c}_i. $$$$ This expression extends Zhou (2012) by showing that the MSPE of the ensembler () depends on the variance of the individual predictor models, their covariance, …”

“…This result has important implications when analysing the epistemic uncertainty of an EN network. Given the zero correlation between the predictions of the sub‐networks (see Mancini et al, 2021 for additional experimental results), then as $$$ T\to \infty $$$, the $$$ \mathrm{MSPE}\left({\overline{f}}_{\mathrm{EN}}\left({\mathbf{x}}_i\right)\right) $$$ converges to zero, assuming that the approximation bias is negligible. Therefore, a suitable prediction interval is $$$$ {\overline{f}}_{\mathrm{EN}}\left({\mathbf{x}}_i\right)\pm {z}_{1-\alpha /2}{\left(\frac{{\hat{\sigma}}_{\hat{\boldsymbol{\omega}}}^2\left({\mathbf{x}}_i\right)}{T}+{\hat{\sigma}}_{\epsilon}^2\right)}^{1/2}, $$$$ with $$$ {\hat{\sigma}}_{\hat{\boldsymbol{\omega}}}^2\left({\mathbf{x}}_i\right)=\frac{1}{T}{\sum}_{t=1}^T{\left({f}_t\left({\mathbf{x}}_i;{\hat{\boldsymbol{\omega}}}^{(t)}\right)-{\overline{f}}_{\mathrm{EN}}\left({\mathbf{x}}_i\right)\right)}^2 $$$...…”

“…Additionally, we propose methods developed in the recent literature on ML to measure the uncertainty around predictions of NNs such as Monte Carlo (MC) dropout, proposed by Gal and Ghahramani (2016), and extra‐neural (EN) networks proposed in Mancini et al (2021). These methods are shown to outperform parametric models such as Hwang and Ding (1997) and non‐parametric bootstrap methods, see Tibshirani (1996).…”