Composite Quantile Regression Neural Network for Massive Datasets

Abstract

Traditional statistical methods and machine learning on massive datasets are challenging owing to limitations of computer primary memory. Composite quantile regression neural network (CQRNN) is an efficient and robust estimation method. But most of existing computational algorithms cannot solve CQRNN for massive datasets reliably and efficiently. In this end, we propose a divide and conquer CQRNN (DC-CQRNN) method to extend CQRNN on massive datasets. The major idea is to divide the overall dataset into some subsets, applying the CQRNN for data within each subsets, and final results through combining these training results via weighted average. It is obvious that the demand for the amount of primary memory can be significantly reduced through our approach, and at the same time, the computational time is also significantly reduced. The Monte Carlo simulation studies and an environmental dataset application verify and illustrate that our proposed approach performs well for CQRNN on massive datasets. The environmental dataset has millions of observations. The proposed DC-CQRNN method has been implemented by Python on Spark system, and it takes 8 minutes to complete the model training, whereas a full dataset CQRNN takes 5.27 hours to get a result.

Spatio-Temporal Forecasting of Global Horizontal Irradiance Using Bayesian Inference

“…is the conditional quantile of y at τ k and θ is a vector of unknown parameters. For a detailed discussion of MCQRNN, see [24,25], among others. QGAM is a hybrid model that is a combination of generalized additive (GAM) and quantile regression (QR) models.…”

Optimal subsampling algorithm for composite quantile regression with distributed data

“…For massive data, Jiang et al (2018) [3] proposed a divide-and-conquer CQR method. Jin and Zhao (2021)[4] proposed a divide-and-conquer CQR neural network method. [13] proposed a distributed CQR method for the massive data.…”