Self-assembled colloidal crystals can exhibit structural colors, a phenomenon of intense reflection within a range of wavelengths caused by constructive interference. Such diffraction effects are most intense for highly uniform crystals; however, in practice, colloidal crystals may include particles of irregular size, which can reduce the quality of the crystal. Despite its importance in realizing high-quality structural colors, a quantitative relationship between particles of irregular size, crystal quality, and the resultant structural color response remains unclear. This study systematically and quantitatively investigates the effect of adding particles of irregular size on the microstructural quality and structural color reflectivity of colloidal crystals formed by evaporative self-assembly via experiment and simulation. We examine two sizes of irregular particlesthose which are 1.9 times larger and 0.4 times smaller than the host crystal. We find that small irregular particles are more detrimental to surface crystal quality and structural color reflectivity than large irregular particles. When incorporated with 10% volume fraction of irregularly sized particles, the reflectivity of crystal films with large (small) irregularly sized particles decreases by 18.4 ± 5.6% (27.5 ± 5.8%), and a measure of surface crystal quality derived from Fourier analysis of scanning electron microscopy images reduces by 40.0 ± 4.5% (48.8 ± 6.0%). By modeling colloidal films incorporated with irregular particles via molecular dynamics simulation and computing the reflection spectra of the modeled crystals via the finite-difference time-domain method, we find that the peak reflectivity of the assembled structures increases monotonically with overall crystallinity, and that overall crystallinity is correlated with the volume fraction of incorporated irregular particles. The quantitative relationships developed in this study can be applied to predict the level of irregularly sized particles that can be tolerated in colloidal films before significant degradation in crystal quality and reflectivity occurs.