Rice blast, rice sheath blight, and rice brown spot have become the most popular diseases in the cold areas of northern China. In order to further improve the accuracy and efficiency of rice disease diagnosis, a framework for automatic classification and recognition of rice diseases is proposed in this study. First, we constructed a training and testing data set including 1,500 images of rice blast, 1,500 images of rice sheath blight, and 1,500 images of rice brown spot, and 1,100 healthy images were collected from the rice experimental field. Second, the deep belief network (DBN) model is designed to include 15 hidden restricted Boltzmann machine layers and a support vector machine (SVM) optimized with switching particle swarm (SPSO). It is noted that the developed DBN and SPSO-SVM can simultaneously learn three proposed features including color, texture, and shape to recognize the disease type from the region of interest obtained by preprocessing the disease images. The proposed model leads to a hit rate of 91.37%, accuracy of 94.03%, and a false measurement rate of 8.63%, with the 10-fold cross-validation strategy. The value of the area under the receiver operating characteristic curve (AUC) is 0.97, whose accuracy is much higher than that of the conventional machine learning model. The simulation results show that the DBN and SPSO-SVM models can effectively extract the image features of rice diseases during recognition, and have good anti-interference and robustness.