To address the automatic detection of dense and small-scale fruit targets under natural large-scene conditions, litchi was used as the research object. Here, a method to automatically detect dense and small-scale litchi fruit targets based on the YOLOv4 detection network is proposed. First, the K-means++ algorithm was used to cluster the labelled data frames (ground truth) to determine the size of the anchor suitable for litchi. Then, the output size of the feature map of the original network was changed to make it more suitable for small-scale target detection. In addition, the images were preprocessed (cropped input) before they were fed into the network. To construct the litchi dataset, 400 images containing more than 20,000 targets were collected. Comparing the detection level to that of the original YOLOv4 model, the recall, precision, and F1 score values of the improved model increased from 0.81 to 0.825, 0.762 to 0.892, and 0.79 to 0.85, respectively. The experimental results indicate that the performance of the litchi detection method proposed in the study is significantly greater than the original model, and it meets the requirements for fruit monitoring in litchi orchards.