Traditional visible light target detection is usually applied in scenes with good visibility, while the advantage of infrared target detection is that it can detect targets at nighttime and in harsh weather, thus being able to be applied to ship detection in complex sea conditions all day long. However, in coastal areas where the density of ships is high and there is a significant difference in target scale, this can lead to missed detection of some dense and small targets. To address this issue, this paper proposes an improved detection model based on YOLOv5s. Firstly, this article designs a feature fusion module based on a fusion attention mechanism to enhance the feature fusion of the network and introduces SPD-Conv to improve the detection accuracy of small targets and low-resolution images. Secondly, by introducing Soft-NMS, the detection accuracy is improved while also addressing the issue of missed detections in dense occlusion situations. Finally, the improved algorithm in this article increased mAP0.5 by 1%, mAP0.75 by 5.7%, and mAP0.5:0.95 by 5% on the infrared ship dataset. A large number of comparative experiments have shown that the improved algorithm in this article is effective at improving detection capabilities.