Vibration-based monitoring and diagnosis provide an excellent and reliable monitoring strategies for maintaining and sustaining a million dollars of industrial assets. The signal processing method is one of the key elements in gearbox fault diagnosis for extracting most useful information from raw vibration signals. Variational mode decomposition (VMD) is one of the recent signal processing methods that helps to solve many limitations in traditional signal processing method. However, pre-determine the input parameters especially the mode number become a challenging task for using this method. Then, this study aims to propose an iterative approach for selecting the mode number for the VMD method by using the normalized mean value (NMV) plot. The NMV value is calculates based on the ratio of a summation of VMD modes and the input signals. The result shows that the proposed iterative VMD approach can select an accurate mode number for the VMD method. Then, the vibration signals decomposed into different VMD modes and used for gearbox fault diagnosis. Statistical features have been extracted from the selected VMD modes and pass into extreme learning machine (ELM) for fault classification. Iterative VMD-ELM provide significance improvement of about 20% higher accuracy in classification result as compared with EMD-ELM. Hence, this research study offers a new mean for gearbox diagnosis strategy.