Timely and accurate acquisition of information on distribution of the crop planting structure in Loess Plateau of eastern Gansu Province, one of the most important agricultural areas in western China, is crucial for promoting fine management of agriculture and ensuring food security. In this study, the Object-Based Image Classification (OBIC), the Random Forest (RF) and Convolutional Neural Network (CNN) models were employed to classify the crop planting structure of four representative test areas in Qingyang City in a precise manner. Firstly, different optimal segmentation scales for various crops were selected using the Estimation of Scale Parameter 2 (ESP2) tool and the Ratio of Mean Difference to Neighbors(ABS) to Standard Deviation (RMAS)model. The images were then segmented through multiresolution segmentation combined with the Canny Edge Detection algorithm. Secondly, the L1 regularized logistic regression model was utilized to select and optimize 39 spatial feature factors including spectral, textural, geometric, and index features, in conjunction with phenological factors. Lastly, under the multi-level classification framework, the Random Forest (RF) classifier and Convolutional Neural Network (CNN) model, combined with object-based multiresolution segmentation, was used to classify the crop planting structure. The findings show that: Thanks to the Canny Edge Detection algorithm, we can obtain a more complete boundary of the segmented objects and improve the separability. The optimal segmentation scales for corn, vegetables, and buckwheat were found to be 55, 70, and 35, respectively, while wheat and apple had optimal segmentation scales of 65. In addition to phenological characteristics, the number of selected spatial features for corn, vegetables, buckwheat, wheat, and apple were 9, 7, 16, 12, and 10, respectively. The CNN model demonstrated high consistency with the RF model in the classification results, but the accuracy of RF model is higher than that of CNN model on the whole. The overall accuracy of classification using RF model in four test areas registered 91.93%, 94.92%, 89.37% and 90.68%, respectively. This paper introduced crop phenological factors, effectively improving the extraction precision of shattered agricultural planting structure in the Loess Plateau of eastern Gansu Province. Its findings have important application value in crop monitoring, management, food security and other related fields.