The quality of fruit is important to increase sales in the market. Right now, the quality selection of orange fruit is mostly still complete by humans. Several drawbacks such as inaccuracy and inconsistency in results happened due to the limitation of human perceptions. The development of computer vision, making it possible to train the computer to classify images based on specified characteristics. This paper proposes the classification model to classify orange images using Convolutional Neural Network (CNN). Five classes of orange namely good-orange-grade-1, good-orange-grade-2, immature-orange, rotten-orange, and damaged-orange are classified using deep learning CNN. Total of 1000 orange images is collected using the smartphone camera. Each class consists of 200 images which are divided into 60% as the training data, 20% as the validation data, and 20% as the testing data. K-Fold Cross-Validation method is used to validate the model. In this paper, the hidden layer of CNN consists of 256 nodes. Two activation functions, ReLU and Tanh, are employed for comparing the accuracy of classification with the Softmax classifier. The result shows that the accuracy of ReLU activation function is 96%, which is better than the Tanh activation function that gives only 93,8%.