In the automatic apple sorting task, it is necessary to automatically classify certain apple species. A shallow convolutional neural network (CNN) architecture is proposed for this purpose. After collecting a certain number of apple images and labelling them, training data is obtained through a series of data augmentation operations, and then training and parameter optimization are carried out through the Caffe framework. The feasibility of the method is verified by experiments which are divided into two cases. In the case of no occlusion, the classification accuracy of apple images reaches approximately 92% in our test set. Besides, block voting is used to aid the proposed method and a good result can be achieved in our test set in the case of part occlusion caused by branches and leaves, rotten spots, and other kinds of apples. The proposed shallow network is characterized by a small number of parameters and shows resistance to overfitting with a limited dataset. Such a network presents an alternative for classification related tasks in smart visual Internet of Things and brings attention to reducing the complexity of deep neural networks while maintaining their strength. INDEX TERMS Image classification, CNN, overfitting, smart visual Internet of Things.