Color is a key indicator in evaluating seed cotton quality. Accurate and rapid detection of seed cotton color is essential for its storage, processing, and trade. In this study, an RGB imaging and semantic segmentation-based method was proposed for seed cotton color detection. First, a color detection system utilizing machine vision technology was developed to capture seed cotton images. Next, a Color-Unet model, incorporating convolutional block attention and improved inception E modules based on Unet, was applied to effectively remove impurities and shadows from the images, resolving the over-segmentation issue commonly encountered in traditional threshold segmentation. The results demonstrated that the pixel accuracy of segmentation reached 97.20%, the mean intersection over union was 91.81%, and the average segmentation speed was 322.3 ms per image. The Color-Unet model effectively addressed the over-segmentation problem. Subsequently, seed cotton color indexes were calculated using Hunter color formulas based on the segmented images. To evaluate the accuracy of color measurement obtained with the proposed method, a regression analysis was performed, comparing the results of those from the HX-410 measurement. The coefficient of determination of yellowness was 0.883, with a root mean square error of 0.150 and a mean relative error of 2.61%. The coefficient of determination of reflectance degree was 0.832, with a root mean square error of 1.56% and a mean relative error of 1.84%. The proposed method allows for the rapid and accurate assessment of seed cotton color from RGB images, providing a valuable technical reference for seed cotton color evaluation.