Since the objects are usually small and densely distributed, the current mainstream algorithms have poor performance in the task of object detection in remote sensing, and are prone to produce missed detection and false detection. Moreover, remote sensing images(RSI) contain complex background noise, which further exacerbates this phenomenon. To settle these problems, this paper proposes a novel small object detection model in remote sensing, named RS-YOLO. In order to suppress the background noise and enhance the attention to small objects, by integrating Biformer, a dynamic attention mechanism, we construct an improved backbone network. Then, an efficient neck structure based on Gather-and-Distribute(GD) mechanism is introduced to enhance the feature fusion capability of the model. In addition, in order to fully utilize shallow features, we simplify the deep layer structure to reduce its competitiveness in the label assignment process. The strategy simultaneously improves the accuracy and speed of the model. Moreover, we combine Inner-IoU and CIoU loss functions as the new bounding box regression(BBR) loss function, which enhances the localization performance and robustness of the model, and speeds up the training time. Finally, on AI-TOD and DIOR datasets, the mean Average Precision (mAP) of our proposed RS-YOLO reaches 56.3\% and 74.2\% respectively, which exceeds the baseline model and other mainstream algorithms.