Aiming at real-time detection of UAVs, small UAV targets are easily missed and difficult to detect in complex backgrounds. To maintain high detection performance while reducing memory and computational costs, this paper proposes the SEB-YOLOv8s detection method. Firstly, the YOLOv8 network structure is reconstructed using SPD-Conv to reduce the computational burden and accelerate the processing speed while retaining more shallow features of small targets. Secondly, we design the AttC2f module and replace the C2f module in the backbone of YOLOv8s with it, enhancing the model’s ability to obtain accurate information and enriching the extracted relevant information. Finally, Bi-Level Routing Attention is introduced to optimize the Neck part of the network, reducing the model’s attention to interfering information and filtering it out. The experimental results show that the mAP50 of the proposed method reaches 90.5% and the accuracy reaches 95.9%, which are improvements of 2.2% and 1.9%, respectively, compared with the original model. The mAP50-95 is improved by 2.7%, and the model’s occupied memory size only increases by 2.5 MB, effectively achieving high-accuracy real-time detection with low memory consumption.