Effectively forecasting energy demand and energy structure helps energy planning departments formulate energy development plans and react to the opportunities and challenges in changing energy demands. In view of the fact that the rolling grey model (RGM) can weaken the randomness of small samples and better present their characteristics, as well as support vector regression (SVR) having good generalization, we propose an ensemble model based on RGM and SVR. Then, the inertia weight of particle swarm optimization (PSO) is adjusted to improve the global search ability of PSO, and the improved PSO algorithm (APSO) is used to assign the adaptive weight to the ensemble model. Finally, in order to solve the problem of accurately predicting the time-series of primary energy consumption, an adaptive inertial weight ensemble model (APSO-RGM-SVR) based on RGM and SVR is constructed. The proposed model can show higher prediction accuracy and better generalization in theory. Experimental results also revealed outperformance of APSO-RGM-SVR compared to single models and unoptimized ensemble models by about 85% and 32%, respectively. In addition, this paper used this new model to forecast China’s primary energy demand and energy structure.