In this paper, we propose an improved single-channel blind source separation (SCBSS) algorithm based on variational modal decomposition (VMD). VMD algorithm is a classical algorithm in SCBSS. The algorithm effectively avoids the end effect and mode mixing, and makes the decomposition more stable. However, the algorithm can not adaptively determine the number of decomposition layers. To solve this problem, we propose an improved adaptive VMD algorithm, which determines the decomposition layer of different signals by setting different spectral correlation coefficient thresholds. Combined with the improved source number estimation algorithm, signal preprocessing algorithm and JADE algorithm, SCBSS of the received mixed signal can be realized. Simulation results show that the algorithm can accurately separate source signals without bit error under different SNR and degrees of spectrum aliasing. The algorithm is effective and feasible.