To achieve accurate detection the content of multiple parts pork adulterated in mutton under the effect of mutton flavor essence and colorant by RGB images, the improved CBAM-Invert-ResNet50 network based on the attention mechanism and the inversion residual was used to detect the content of pork from the back, front leg, and hind leg in adulterated mutton. The deep features of different parts extracted by the CBAM-Invert-ResNet50 were fused by feature, stitched, and combined with transfer learning, and the content of pork from mixed parts in adulterated mutton was detected. The results showed that the R2 of the CBAM-Invert-ResNet50 for the back, front leg, and hind leg datasets were 0.9373, 0.8876, and 0.9055, respectively, and the RMSE values were 0.0268 g·g−1, 0.0378 g·g−1, and 0.0316 g·g−1, respectively. The R2 and RMSE of the mixed dataset were 0.9264 and 0.0290 g·g−1, respectively. When the features of different parts were fused, the R2 and RMSE of the CBAM-Invert-ResNet50 for the mixed dataset were 0.9589 and 0.0220 g·g−1, respectively. Compared with the model built before feature fusion, the R2 of the mixed dataset increased by 0.0325, and the RMSE decreased by 0.0070 g·g−1. The above results indicated that the CBAM-Invert-ResNet50 model could effectively detect the content of pork from different parts in adulterated mutton as additives. Feature fusion combined with transfer learning can effectively improve the detection accuracy for the content of mixed parts of pork in adulterated mutton. The results of this study can provide technical support and a basis for maintaining the mutton market order and protecting mutton food safety supervision.