Investigating the interactions between universal and culturally specific influences on color categorization across individuals and cultures has proven to be a challenge for human color categorization and naming research. The present article simulates the evolution of color lexicons to evaluate the role of two realistic constraints found in the human phenomenon: (i) heterogeneous observer populations and (ii) heterogeneous color stimuli. Such constraints, idealized and implemented as agent categorization and communication games, produce interesting and unexpected consequences for stable categorization solutions evolved and shared by agent populations. We find that the presence of a small fraction of color deficient agents in a population, or the presence of a "region of increased salience" in the color stimulus space, break rotational symmetry in population categorization solutions, and confine color category boundaries to a subset of available locations. Further, these heterogeneities, each in a different, predictable, way, might lead to a change of category number and size. In addition, the concurrent presence of both types of heterogeneity gives rise to novel constrained solutions which optimize the success rate of categorization and communication games. Implications of these agent-based results for psychological theories of color categorization and the evolution of color naming systems in human societies are discussed.