Indonesia's agricultural production is divided into two main categories: vegetables and fruits. The vegetable category includes shallots, garlic, chilies, mushrooms, spinach, cabbage, and potatoes. One of the fruit commodities from the fruit horticulture subsector is bananas, which are divided into several types, including ambon, plantains, Cavendish, pipit, and horn bananas. One of the bananas that has a good selling value in Indonesia is the Cavendish banana, but the selling value of the Cavendish banana is determined by the quality of the banana fruit. A classification process is necessary to find out the quality of bananas. We perform classification using one of the deep learning algorithms, namely Convolutional Neural Network. The experiment uses 1047 images, divided into 65% training data, 15% validation data, and 20% testing data by using epochs 20 times with 16 batch sizes, the accurate results obtained are 99%. The results indicate the effectiveness of the confusion matrix in identifying training data and detecting images. It can be concluded that using more training data leads to higher accuracy, as fewer image reading errors occur when fewer images are processed. This classification is expected to be able to classify bananas with good quality like the real condition.