The development of automated grading equipment requires achieving high throughput and precise detection of disease spots on jujubes. However, the current algorithms are inadequate in accomplishing these objectives due to their high density, varying sizes and shapes, and limited location information regarding disease spots on jujubes. This paper proposes a method called JujubeSSD, to boost the precision of identifying disease spots in jujubes based on a single shot multi-box detector (SSD) network. In this study, a diverse dataset comprising disease spots of varied sizes and shapes, varying densities, and multiple location details on jujubes was created through artificial collection and data augmentation. The parameter information obtained from transfer learning into the backbone feature extraction network of the SSD model, which reduced the time of spot detection to 0.14 s. To enhance the learning of target detail features and improve the recognition of weak information, the traditional convolution layer was replaced with deformable convolutional networks (DCNs). Furthermore, to address the challenge of varying sizes and shapes of disease spot regions on jujubes, the path aggregation feature pyramid network (PAFPN) and balanced feature pyramid (BFP) were integrated into the SSD network. Experimental results demonstrate that the mean average precision at the IoU (intersection over union) threshold of 0.5 (mAP@0.5) of JujubeSSD reached 97.1%, representing an improvement of approximately 6.35% compared to the original algorithm. When compared to existing algorithms, such as YOLOv5 and Faster R-CNN, the improvements in mAP@0.5 were 16.84% and 8.61%, respectively. Therefore, the proposed method for detecting jujube disease spot achieves superior performance in jujube surface disease detection and meets the requirements for practical application in agricultural production.