Variational Mode Decomposition (VMD) is a signal decomposition algorithm with excellent denoising ability. However, the drawback that VMD is unable to determine the input parameters adaptively seriously affects the decomposition results. For this issue, an optimized VMD method based on modified scale-space representation (MSSR-VMD) is proposed. Firstly, MSSR is proposed to segment the fault signal spectrum, acquiring modes' number and the initial center frequency for each mode adaptively. Moreover, a pre-decomposition step is added to the original VMD, which selects a target mode from divided frequency bands. Finally, the penalty factor of the target mode is adjusted during the iterative update of the VMD to achieve accurate extraction for the fault features. MSSR-VMD and other adaptive decomposition algorithms are employed to handle the simulated and experimental signals separately. By comparing the analysis results, the method has certain superiority in rolling bearing fault feature extraction.