To improve the model's efficiency, people use many different methods, including the Transfer Learning algorithm, to improve the efficiency of recognition and classification of image data. The study was carried out to combine optimization algorithms with the Transfer Learning model with MobileNet, MobileNetV2, InceptionV3, Xception, ResNet50V2, DenseNet201 models. Then, testing on rice disease data set with 13.186 images, background removed. The result obtained with high accuracy is the RMSprop algorithm, with an accuracy of 88% when combined with the Xception model, similar to the F1, Xception model, and ResNet50V2 score of 87% when combined with the Adam algorithm. This shows the effect of gradients on the Transition learning model. Research, evaluate and draw the optimal model to build a website to identify diseases on rice leaves, with the main functions including images and recording of disease identification points for better management purposes on diseased areas of rice.