Indonesia is the fourth biggest rice producer in Asia with its production accounting for 35.4 million metric tons yearly. This figure can increase unless rice crop failure is resolved. Identifying rice diseases, however, may serve as an approach to minimizing the risk of crop failure. The classification to detect rice diseases was previously researched using ResNet101 method with 100% accuracy. Despite this perfect accuracy, this approach does not come without an issue, where the prediction is not yet optimal for each label and loss results which are regarded as too high due to overfitting. Departing from this issue, this research aims to improve the model by reducing the layer complexity of the model and comparing two layers structures of the model, two different data, and the ResNet101 model. The performance resulting from the model could be enhanced with the structuring of simple architectural layers. Despite the small quantity of dataset, the model performance can yield 100% accuracy in the classification of rice diseases with a loss value of 2.91%. The model performance in this research experienced a 2.7% increase at the loss value and it could accurately classify the type of rice diseases according to leaf images on each label. The problem solved by this research is that ResNet101 is able to classify rice disease accurately even with a small amount of data by utilizing the appropriate layer arrangement with data requirements. In addition, the overfitting that occurred in previous research can also be resolved properly. This matter proves that the correlation between the layers of the model with the amount of data is very influential.