Gearboxes are massively utilized in nowadays industries due to their huge importance in power transmission; hence, their defects can heavily affect the machines performance. Therefore, many researchers are working on gearboxes fault detection and classification. However, most of the works are carried out under constant speed conditions, while gears usually operate under varying speed and torque conditions, making the task more challenging. In this paper, we propose a new method for gearboxes condition monitoring that is efficiently able to reveal the fault from the vibration signatures under varying operating condition. First, the vibration signal is processed with the Maximal Overlap Discrete Wavelet Packet Transform (MODWPT) to extract the AM-FM modes. Next, time domain features are calculated from each mode. Then the features set are reduced using the Ant colony optimization algorithm (ACO) by removing the redundant and unimportant parameters that may mislead the classification. Finally, an ensemble learning algorithm Random Forest (RF) is used to train a model able to classify the fault based on the selected features. The innovative aspect about this method is that, unlike other existing methods, ACO is able to optimize not only the features but also the parameters of the classifier in order to obtain the highest classification accuracy. The proposed method was tested on varying operating condition real dataset consisting of six different gearboxes. In the aim to prove the performance of our method, it had been compared to other conventional methods. The obtained results indicate its robustness, and its accuracy stability to handle the varying operating condition issue in gearboxes fault detection and classification with high efficiency.