Due to the complicated structure, vibration signal of rotating machinery is multicomponent with nonstationary and nonlinear features, so it is difficult to diagnose faults effectively. Therefore, effective extraction of vibration signal characteristics is the key to diagnose the faults of rotating machinery. Mode mixing and illusive components existed in some conventional methods, such as EMD and EEMD, which leads to misdiagnosis in extracting signals. Given these reasons, a new fault diagnosis method, namely, variation mode decomposition (VMD), was proposed in this paper. VMD is a newly developed technique for adaptive signal decomposition, which can decompose a multicomponent signal into a series of quasi-orthogonal intrinsic mode functions (IMFs) simultaneously, corresponding to the components of signal clearly. To further research on VMD method, the advantages and characteristics of VMD are investigated via numerical simulations. VMD is then applied to detect oil whirl and oil whip for rotor systems fault diagnosis via practical vibration signal. The experimental results demonstrate the effectiveness of VMD method.