Foreign fibers directly impact the quality of raw cotton, affecting the prices of textile products and the economic efficiency of cotton textile enterprises. The accurate differentiation and labeling of foreign fibers require domain-specific knowledge, and labeling scattered cotton foreign fibers in images consumes substantial time and labor costs. In this study, we propose a semi-supervised foreign fiber detection approach that uses unlabeled image information and a small amount of labeled data for model training. Our proposed method, Efficient YOLOv5-cotton, introduces CBAM to address the issue of the missed detection and false detection of small-sized cotton foreign fibers against complex backgrounds. Second, the algorithm designs a multiscale feature information extraction network, SPPFCSPC, which improves its ability to generalize to fibers of different shapes. Lastly, to reduce the increased network parameters and computational complexity introduced by the SPPFCSPC module, we replace the C3 layer with the C3Ghost module. We evaluate Efficient YOLOv5 for detecting various types of foreign fibers. The results demonstrate that the improved Efficient YOLOv5-cotton achieves a 1.6% increase in mAP@0.5 (mean average precision) compared with the original Efficient YOLOv5 and reduces model parameters by 10% compared to the original Efficient YOLOv5 with SPPFCSPC. Our experiments show that our proposed method enhances the accuracy of foreign fiber detection using Efficient YOLOv5-cotton and considers the trade-off between the model size and computational cost.