Vibration signal analysis is an efficient online transformer fault diagnosis method for improving the stability and safety of power systems. Operation in harsh interference environments and the lack of fault samples are the most challenging aspects of transformer fault diagnosis. High-precision performance is difficult to achieve when using conventional fault diagnosis methods. Thus, this study proposes a transformer fault diagnosis method based on the adaptive transfer learning of a two-stream densely connected residual shrinkage network over vibration signals. First, novel time-frequency analysis methods (i.e., Synchrosqueezed Wavelet Transform and Synchrosqueezed Generalized S-transform) are proposed to convert vibration signals into different images, effectively expanding the samples and extracting effective features of signals. Second, a Two-stream Densely Connected Residual Shrinkage (TSDen2NetRS) network is presented to achieve a high accuracy fault diagnosis under different working conditions. Furthermore, the Residual Shrinkage layer (RS layer) is applied as a nonlinear transformation layer to the deep learning framework to remove unimportant features and enhance anti-interference performance. Lastly, an adaptive transfer learning algorithm that can automatically select the source data set by using the domain measurement method is proposed. This algorithm accelerates the training of the deep learning network and improves accuracy when the number of samples is small. Vibration experiments of transformers are conducted under different operating conditions, and their results show the effectiveness and robustness of the proposed method.