The capacity configuration of the energy storage system plays a crucial role in enhancing the reliability of the power supply, power quality, and renewable energy utilization in microgrids. Based on variational mode decomposition (VMD), a capacity optimization configuration model for a hybrid energy storage system (HESS) consisting of batteries and supercapacitors is established to achieve the optimal configuration of energy storage capacity in wind–solar complementary islanded microgrids. Firstly, based on the energy mapping relationship between the frequency domain and time domain, the decomposition mode number K of VMD is determined based on the principle of minimum total mode aliasing energy. Then, considering the smoothing fluctuation characteristics of different energy storage components, the dividing point N of high frequency and low frequency in the unbalanced power between the source and load in the microgrid is selected to allocate charging and discharging power instructions for the battery and supercapacitor. Finally, taking the annual comprehensive cost of the HESS as the objective function, a hybrid energy storage capacity optimization configuration model is established, and the dividing point N is used as the optimization variable to solve the model in order to obtain the optimal configuration results. The case study results show that the proposed method is more economical and feasible than the existing energy storage configuration methods.