The plum is a kind of delicious and common fruit with high edible value and nutritional value. The accurate and effective detection of plum fruit is the key to fruit number counting and pest and disease early warning. However, the actual plum orchard environment is complex, and the detection of plum fruits has many problems, such as leaf shading and fruit overlapping. The traditional method of manually estimating the number of fruits and the presence of pests and diseases used in the plum growing industry has disadvantages, such as low efficiency, a high cost, and low accuracy. To detect plum fruits quickly and accurately in a complex orchard environment, this paper proposes an efficient plum fruit detection model based on an improved You Only Look Once version 7(YOLOv7). First, different devices were used to capture high-resolution images of plum fruits growing under natural conditions in a plum orchard in Gulin County, Sichuan Province, and a dataset for plum fruit detection was formed after the manual screening, data enhancement, and annotation. Based on the dataset, this paper chose YOLOv7 as the base model, introduced the Convolutional Block Attention Module (CBAM) attention mechanism in YOLOv7, used Cross Stage Partial Spatial Pyramid Pooling–Fast (CSPSPPF) instead of Cross Stage Partial Spatial Pyramid Pooling(CSPSPP) in the network, and used bilinear interpolation to replace the nearest neighbor interpolation in the original network upsampling module to form the improved target detection algorithm YOLOv7-plum. The tested YOLOv7-plum model achieved an average precision (AP) value of 94.91%, which was a 2.03% improvement compared to the YOLOv7 model. In order to verify the effectiveness of the YOLOv7-plum algorithm, this paper evaluated the performance of the algorithm through ablation experiments, statistical analysis, etc. The experimental results showed that the method proposed in this study could better achieve plum fruit detection in complex backgrounds, which helped to promote the development of intelligent cultivation in the plum industry.