Comparison of universal approximators incorporating partial monotonicity by structure

Minin, Alexey; Velikova, Marina; Lang, Bernhard; Daniels, Hennie

doi:10.1016/j.neunet.2009.09.002

Cited by 22 publications

(16 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, extra hidden layers of positive parameters can be added to the model. As pointed out by Lang (2005) and Minin et al (2010), an additional hidden layer is required for the MMLP to maintain its universal function approximation capabilities. While multiple hidden layers are included in the software implementation by Cannon (2017), for sake of simplicity, this study only considers the single hidden layer architecture of Zhang and Zhang (1999).…”

Section: Monotone Multi-layer Perceptron (Mmlp)mentioning

confidence: 99%

“…These features, which are combined into a single, unified framework, are made possible through a novel combination of elements drawn from the standard QRNN model (White 1992;Taylor 2000;Cannon 2011), the monotone multi-layer perceptron (MMLP) (Zhang and Zhang 1999;Lang 2005;Minin et al 2010), the composite QRNN (CQRNN) , the expectile regression neural network , and the generalized additive neural network (Potts 1999). To the best of the author's knowledge, the MCQRNN model is the first neural networkbased implementation of quantile regression that guarantees non-crossing of regression quantiles.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes

Cannon

2018

Stoch Environ Res Risk Assess

103

View full text Add to dashboard Cite

The goal of quantile regression is to estimate conditional quantiles for specified values of quantile probability using linear or nonlinear regression equations. These estimates are prone to ''quantile crossing'', where regression predictions for different quantile probabilities do not increase as probability increases. In the context of the environmental sciences, this could, for example, lead to estimates of the magnitude of a 10-year return period rainstorm that exceed the 20-year storm, or similar nonphysical results. This problem, as well as the potential for overfitting, is exacerbated for small to moderate sample sizes and for nonlinear quantile regression models. As a remedy, this study introduces a novel nonlinear quantile regression model, the monotone composite quantile regression neural network (MCQRNN), that (1) simultaneously estimates multiple non-crossing, nonlinear conditional quantile functions; (2) allows for optional monotonicity, positivity/ non-negativity, and generalized additive model constraints; and (3) can be adapted to estimate standard least-squares regression and non-crossing expectile regression functions. First, the MCQRNN model is evaluated on synthetic data from multiple functions and error distributions using Monte Carlo simulations. MCQRNN outperforms the benchmark models, especially for non-normal error distributions. Next, the MCQRNN model is applied to real-world climate data by estimating rainfall Intensity-Duration-Frequency (IDF) curves at locations in Canada. IDF curves summarize the relationship between the intensity and occurrence frequency of extreme rainfall over storm durations ranging from minutes to a day. Because annual maximum rainfall intensity is a non-negative quantity that should increase monotonically as the occurrence frequency and storm duration decrease, monotonicity and non-negativity constraints are key constraints in IDF curve estimation. In comparison to standard QRNN models, the ability of the MCQRNN model to incorporate these constraints, in addition to non-crossing, leads to more robust and realistic estimates of extreme rainfall.

show abstract

Section: Monotone Multi-layer Perceptron (Mmlp)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes

Cannon

2018

Stoch Environ Res Risk Assess

103

View full text Add to dashboard Cite

show abstract

“…Archer and Wang [3] propose a monotone model by constraining the neural net weights to be positive. Other methods enforce constraints on model weights [11,46,33,16,2], and force the derivative of the output to be strictly positive [47]. Monotonic networks [40] guarantee monotonicity by constructing a threelayer network using monotonic linear embedding and max-min-pooling.…”

Section: Related Workmentioning

confidence: 99%

Counterexample-Guided Learning of Monotonic Neural Networks

Sivaraman¹,

Farnadi²,

Millstein³

et al. 2020

Preprint

View full text Add to dashboard Cite

The widespread adoption of deep learning is often attributed to its automatic feature construction with minimal inductive bias. However, in many real-world tasks, the learned function is intended to satisfy domain-specific constraints. We focus on monotonicity constraints, which are common and require that the function's output increases with increasing values of specific input features. We develop a counterexample-guided technique to provably enforce monotonicity constraints at prediction time. Additionally, we propose a technique to use monotonicity as an inductive bias for deep learning. It works by iteratively incorporating monotonicity counterexamples in the learning process. Contrary to prior work in monotonic learning, we target general ReLU neural networks and do not further restrict the hypothesis space. We have implemented these techniques in a tool called COMET. 1 Experiments on real-world datasets demonstrate that our approach achieves state-of-the-art results compared to existing monotonic learners, and can improve the model quality compared to those that were trained without taking monotonicity constraints into account.

show abstract

“…This method estimates simultaneously multiple non-crossing quantile functions and allows for optional monotonicity, positivity/non-negativity, and additivity constraints. Therefore, it combines elements drawn from the standard QRNN model [5,33,34], the monotone multi-layer perceptron (MMLP) [38,39] and the composite QRNN (CQRNN) [40]. The basis of the MCQRNN is the multi-layer perceptron (MLP) neural network with partial monotonicity constraints [41].…”

Section: Quantile Regression Neural Network (Qrnn)mentioning

confidence: 99%

Machine Learning Techniques for Predicting the Energy Consumption/Production and Its Uncertainties Driven by Meteorological Observations and Forecasts

2019

View full text Add to dashboard Cite

Reliable predictions of the energy consumption and production is important information for the management and integration of renewable energy sources. Several different Machine Learning (ML) methodologies have been tested for predicting the energy consumption/production based on the information of hydro-meteorological data. The methods analysed include Multivariate Adaptive Regression Splines (MARS) and various Quantile Regression (QR) models like Quantile Random Forest (QRF) and Gradient Boosting Machines (GBM). Additionally, a Nonhomogeneous Gaussian Regression (NGR) approach has been tested for combining and calibrating monthly ML based forecasts driven by ensemble weather forecasts. The novelty and main focus of this study is the comparison of the capability of ML methods for producing reliable predictive uncertainties and the application of monthly weather forecasts. Different skill scores have been used to verify the predictions and their uncertainties and first results for combining the ML methods applying the NGR approach and coupling the predictions with monthly ensemble weather forecasts are shown for the southern Switzerland (Canton of Ticino). These results highlight the possibilities of improvements using ML methods and the importance of optimally combining different ML methods for achieving more accurate estimates of future energy consumptions and productions with sharper prediction uncertainty estimates (i.e., narrower prediction intervals).

show abstract

Comparison of universal approximators incorporating partial monotonicity by structure

Cited by 22 publications

References 5 publications

Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes

Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes

Counterexample-Guided Learning of Monotonic Neural Networks

Machine Learning Techniques for Predicting the Energy Consumption/Production and Its Uncertainties Driven by Meteorological Observations and Forecasts

Contact Info

Product

Resources

About