Potatoes are short-term crops grown for harvesting tubers. It is a type of tuber that grows on roots and is the fourth most common crop after rice, wheat, and corn. Fresh potatoes can also be used in an incredible variety of dishes by baking, boiling, or frying them. Moreover, the paper, textile, wood, and pharmaceutical industries also make extensive use of potato starch. However, soil and climate pollution are highly unfavorable for potato growth and lead to a lot of diseases such as common scab, black scurf, blackleg, dry rot, and pink rot. Thus. several types of research in medicine and computers were started for the early detection, classification, and treatment of potato diseases. In this study, transfer learning and finetuning were applied to potato disease classification based on a custom ConvNeXtSmall model. In addition, Gradient-weighted Class Activation Mapping (i.e., Grad-CAM) is provided for visual explanation in the final result after classification. For potato illness segmentation, k-means clustering was used to enable the difference between healthy and diseased sections based on color and texture. The data was collected from numerous websites and validated by the Bangladesh Agricultural Research Institute (i.e., BARI), including six types of potato diseases and healthy images. With a Convolutional Neural Networks (i.e., CNN) model from the Keras library, our study reached the unexpected validation accuracy, test accuracy, and F1 score in seven classifications of 99.49%, 98.97%, and 98.97%, respectively. Concerning four-class classification, high accuracy values were obtained for most of the models (i.e., 100%).