Diabetic retinopathy (DR) is an essential factor that has caused vision loss and even blindness in middle-aged and older adults. A system that can automatically perform DR diagnosis can help ophthalmologists save a lot of tedious work, such as DR grading or lesion detection. At the same time, patients can find their diseases earlier and perform the correct treatment. However, most of the existing methods require many DR annotations to train the model, and the DR data will vary to different degrees due to various shooting tools. The above problems lead to the inefficient use of existing data in the experiment, limiting actual deployment. To alleviate this problem, we propose a novel Graph Adversarial Transfer Learning (GATL) for DR diagnosis in a deep model through transfer learning, including intradomain alignment and inter-domain alignment. The proposed GATL enjoys several merits. First, our GATL adopts the self-supervised training to save the annotating cost in the target domain thus this domain adaptation method can significantly reduce annotation cost compared to the supervised approaches. Second, we introduce the graph neural network to extract potential features between unknown samples. Third, to enhance the robustness of the model, we use adversarial training to perform both inter-domain and intradomain alignment to further improve the model's classification accuracy. GATL achieved 94.3%, 97.5%, and 91.1% in accuracy, sensitivity, and specificity in the APTOS dataset and 92.7%, 95.7%, and 89.7% in the EyePACS dataset, respectively. Extensive experimental results on two challenging benchmarks, including APTOS 2019 and EyePACS, demonstrate that the proposed GATL performs favorably against baseline DR classification methods.