Sequential Data Processing for IMERG Satellite Rainfall Comparison and Improvement Using LSTM and ADAM Optimizer

Abstract

This study introduces a systematic methodology whereby different technologies were utilized to download, pre-process, and interactively compare the rainfall datasets from the Integrated Multi-Satellite Retrievals for Global Precipitation Mission (IMERG) satellite and rain gauges. To efficiently handle the large volume of data, we developed automated shell scripts for downloading IMERG data and storing it, along with rain gauge data, in a relational database system. Hypertext pre-processor (pHp) programs were built to visualize the result for better analysis. In this study, the performance of IMERG estimations over the east coast of Peninsular Malaysia for the duration of 10 years (2011–2020) against rain gauge observation data is evaluated. Moreover, this study aimed to improve the daily IMERG estimations with long short-term memory (LSTM) developed with Python. Findings show that the LSTM with Adaptive Moment Estimation (ADAM) optimizer trained against the mean square error (MSE) loss enhances the accuracy of satellite estimations. At the point-to-pixel scale, the correlation between satellite estimations and ground observations was increased by about 15%. The bias was reduced by 81–118%, MAE was reduced by 18–59%, the root-mean-square error (RMSE) was reduced by 1–66%, and the Kling–Gupta efficiency (KGE) was increased by approximately 200%. The approach developed in this study establishes a comprehensive and scalable data processing and analysis pipeline that can be applied to diverse datasets and regions encountering similar domain-specific challenges.

Towards Optimal Solar Energy Integration: A Deep Dive into AI-Enhanced Solar Irradiance Forecasting Models

“…The rationale behind its adoption is the optimizer's renowned capability for adaptive learning rates for each parameter. By leveraging moment estimates of the gradients, ADAM provides a more sophisticated and efficient trajectory in the parameter space, thereby potentially accelerating convergence [78,79]. Known for its effectiveness, this optimizer significantly enhances the model's precision.…”

Personalized Point-of-Interest Recommendation Using Improved Graph Convolutional Network in Location-Based Social Network

“…After tuning the parameters through cross-validation, set the batch size to 2048, the negative sampling rate to 300, the learning rate lr = 10 −3 , the regularization coefficient λ = 10 −4 , the weight coefficient w = 0.5, α = 1.1, β = 1, and the hyperparameter w 1 = 1, w 2 = 0.01, and w 3 = 0.01. The Adam [36] algorithm is used to optimize the model parameters. The weight coefficients η 1 and η 2 of the higher-order social friend influence and community neighbor influence can be learned in the training model.…”