Intelligent sorting of coal and gangue is of great significance to the intelligent construction of coal mines as well as green development. In this study, we propose a coal and gangue segmentation method with an improved classical segmentation network Mask R‐CNN, denoted as Multichannel Forward‐Linked Confusion Convolution Module (MFCCM)‐Mask R‐CNN. First, we design a MFCCM to construct the feature extraction network by stacking, second, we design a multiscale high‐resolution feature pyramid network structure to realize multipath fusion of feature information to enhance the position and contour information of the target, and finally, we propose a multiscale Mask head to enhance the diversity of information, and capture the more representative and unique features. Training and testing models using self‐built RGB coal and gangue data sets, the accuracy of the improved algorithm reaches 97.38%, which is an improvement of 1.66% compared to the original model. Compared with other segmentation models Unet, Deeplab V3+, Yoloact, Yolov7, and the model after replacing the backbone network, the MFCCM‐Mask R‐CNN has higher precision and recall, and can more accurately realize the efficient segmentation of coal and gangue.