Bearings are one of the critical components of rotating machinery, and their failure can cause catastrophic consequences. In this regard, previous studies have proposed a variety of intelligent diagnosis methods. Most existing bearing fault diagnosis methods implicitly assume that the training and test sets are from the same distribution. However, in real scenarios, bearings have been working in complex and changeable working environments for a long time. The data during their working processes and the data used for model training cannot meet this condition. This paper proposes an improved adversarial transfer network for fault diagnosis under variable working conditions. Specifically, this paper combines an adversarial transfer network with a short-time Fourier transform to obtain satisfactory results with the lighter network. Then, this paper employs a channel attention module to enhance feature fusion. Moreover, this paper designs a novel domain discrepancy hybrid metric loss to improve model transfer learning performance. Finally, this paper verifies the method’s effectiveness on three datasets, including dual-rotor, a Case Western Reserve University dataset and the Ottawa dataset. The proposed method achieves average accuracy, surpassing other methods, and shows better domain alignment capabilities.