There is a complex interaction between primary fibers in pre-colored fiber blends, which leads to the impracticality of Duncan’s additive theory in the single-constant Kubelka–Munk (KM-1) model, resulting in poor accuracy for color prediction. This paper builds an optimized combined spectral calibration model based on the KM-1 function to solve this issue. This proposed model involves colored and non-colored parts with trained spectral coefficients, and was established to achieve a good matching between the predicted and actual ratio of the absorption coefficient K to the scattering coefficient S, K/ S of samples for color prediction from the perspective of the spectrum. Experimentally, five primary cotton fibers were selected and prepared into fiber blends to demonstrate its validation, and its optimal number of training samples was found to be only 14, which was much less than the general training sampling method using half of all samples directly. Finally, a remarkable average color difference of only 0.63 CIEDE2000 units for 74 test specimens was achieved on this optimized model, which was significantly lower than that of the KM-1 model (∼1.16), two-constant KM model (∼1.03), Stearns–Noechel model (∼1.16) and Friele model (∼1.11). The results indicate that the optimized model behaves well and could be applied in the color prediction of pre-colored fiber blends.