Coal gangue sorting is a necessary process in coal mine production, and removing gangue is the basis for the coal production of clean energy; it is also an important approach to reduce the cost of washing, improve the grade of finished coal and increase the economic efficiency of coal mining enterprises. For the problem of high similarity and low-degree dynamic recognition of coal and gangue, a coal gangue target detection method based on improved YOLOv5s is proposed. Based on the YOLOv5s network, the decoupled head and SimAM attention mechanism are introduced and the CSP module in the neck part of YOLOv5s is replaced with the VoV-GSCSP structure. The experimental results show that the proposed method improves the mAP value by 6.1% over YOLOv5s in the gangue target detection task, while maintaining a higher detection speed. The coal gangue classification precision reaches 99.7% when tested on 1479 images. Compared with YOLOv5 series, YOLOv7 series, SSD and Faster-RCNN, the proposed method invariably yields higher precision and detection speed to meet the requirements of real-time detection. The experiments prove that the method proposed in this paper can be applied to the coal gangue sorting industry for fast and high-precision identification of coal gangue.