Given the wide applications of photovoltaic (PV) power generation, the volatility in generation caused by solar radiation, which limits the capacity of the power grid, cannot be ignored. Therefore, much research has aimed to address this issue through the development of methods for accurately predicting inter-hour solar radiation and then estimating PV power. However, most forecasting methods focus on adjusting the model structure or model parameters to achieve prediction accuracy. There is little research discussing how different factors influence solar radiation and, thereby, the effectiveness of these data-driven methods regarding their prediction accuracy. In this work, the effects of several potential factors on solar radiation are estimated using correlation analysis and a structural equation model; an ensemble model is developed for predicting inter-hour solar radiation based on the interaction of those key factors. Several experiments are carried out based on an open database provided by the National Renewable Energy Laboratory. The results show that solar zenith angle, cloud cover, aerosols, and airmass have great effects on solar radiation. It is also shown that the selection of the key factor is more important than the model structure construction for predicting solar radiation precisely. The proposed ensemble model proves to outperform all sub-models and achieves about a 12% improvement over the persistent model based on the normalized root mean squared error statistic.