X-ray images typically contain complex background information and abundant small objects, posing significant challenges for object detection in security tasks. Most existing object detection methods rely on complex networks and high computational costs, which poses a challenge to implement lightweight models. This article proposes Fine-YOLO to achieve rapid and accurate detection in the security domain. First, a low-parameter feature aggregation (LPFA) structure is designed for the backbone feature network of YOLOv7 to enhance its ability to learn more information with a lighter structure. Second, a high-density feature aggregation (HDFA) structure is proposed to solve the problem of loss of local details and deep location information caused by the necked feature fusion network in YOLOv7-Tiny-SiLU, connecting cross-level features through max-pooling. Third, the Normalized Wasserstein Distance (NWD) method is employed to alleviate the convergence complexity resulting from the extreme sensitivity of bounding box regression to small objects. The proposed Fine-YOLO model is evaluated on the EDS dataset, achieving a detection accuracy of 58.3% with only 16.1 M parameters. In addition, an auxiliary validation is performed on the NEU-DET dataset, the detection accuracy reaches 73.1%. Experimental results show that Fine-YOLO is not only suitable for security, but can also be extended to other inspection areas.