Solar flares are solar storm events driven by the magnetic field in the solar activity area. Solar flare, often associated with solar proton event or CME, has a negative impact on ratio communication, aviation, and aerospace. Therefore, its forecasting has attracted much attention from the academic community. Due to the limitation of the unbalanced distribution of the observation data, most techniques failed to effectively learn complex magnetic field characteristics, leading to poor forecasting performance. Through the statistical analysis of solar flare magnetic map data observed by SDO/HMI from 2010 to 2019, we find that unsupervised clustering algorithms have high accuracy in identifying the sunspot group in which the positive samples account for the majority. Furthermore, for these identified sunspot groups, the ensemble model that integrates the capability of boosting and convolutional neural network (CNN) achieves high-precision prediction of whether the solar flares will occur in the next 48 hours. Based on the above findings, a two-stage solar flare early warning system is established in this paper. The F1 score of our method is 0.5639, which shows that it is superior to the traditional methods such as logistic regression and support vector machine (SVM).