In the spectrum measurement experiment, the roughness of the object surface is an essential factor that cannot be ignored. In this experiment, a group of mixed pixel samples with different mixing ratios were designed, and these samples were printed on four kinds of papers with different roughness. The spectral characteristics of mixed pixels with different roughness are quantitatively analyzed by using the measured spectral data. The linear spectral mixture model is used for spectral decomposition, and the effect of roughness on the unmixing precision of mixed pixels was studied. The surface roughness will affect the reflectivity of the mixed pixel. Specifically, the higher the roughness is, the higher the reflectivity of the sample is. This phenomenon is more noticeable when the proportion of white endmember (PWE) is large, and as the white area ratio decreases, the reflectance difference gradually decreases. When the surface roughness of the sample is less than 3.339 μm, the spectral decomposition is performed using a linear spectral mixing model in the visible light band. The average error of the unmixing is less than 0.53%, which is lower than the conventional standard spectral measurement error. In other words, when the surface roughness of the sample is controlled within a specific range, the effect of roughness on the unmixing accuracy of the mixed pixels is small, and this effect can be almost ignored. Multiple scattering within the pixels is the key to model selection and unmixing accuracy, when using the ASD FieldSpec3 spectrometer to perform spectral reflectance measurement and linear spectral unmixing experiments. If the surface roughness of the sample to be measured is less than the maximum wavelength of the spectrometer, the experimental results believe that the photon energy is mainly mirror reflection on the surface of the object and diffuse reflection. At this time, it is still a better choice to use a linear spectral mixing model to decompose the mixed pixels.