Small object detection poses significant challenges in the realm of general object detection, primarily due to complex backgrounds and other instances interfering with the expression of features. This research introduces an uncomplicated and efficient algorithm that addresses the limitations of small object detection. Firstly, we propose an efficient cross-scale feature fusion attention module called ECFA, which effectively utilizes attention mechanisms to emphasize relevant features across adjacent scales and suppress irrelevant noise, tackling issues of feature redundancy and insufficient representation of small objects. Secondly, we design a highly efficient convolutional module named SEConv, which reduces computational redundancy while providing a multi-scale receptive field to improve feature learning. Additionally, we develop a novel dynamic focus sample weighting function called DFSLoss, which allows the model to focus on learning from both normal and challenging samples, effectively addressing the problem of imbalanced difficulty levels among samples. Moreover, we introduce Wise-IoU to address the impact of poor-quality examples on model convergence. We extensively conduct experiments on four publicly available datasets to showcase the exceptional performance of our method in comparison to state-of-the-art object detectors.