Color appearance models are usually based on equations with fixed numerical coefficients that were predetermined to yield optimal agreement for a selected standard observer. Here we address the well‐known fact that, among color‐normal observers, there are significant differences of cone fundamentals, which often cause observer metamerism. This is not fully addressed by many appearance models. It might seem that observer metamerism could be accurately predicted by substituting an individual's color matching functions into an otherwise‐unchanged color appearance model, but this is not assured because the model's coefficients were not designed for the new observer. Here we explore a solution related to how the visual system tends to compensate for changes in color information arising from variations cone fundamentals. For this purpose, we developed a simple color appearance model that uses only a few numerical coefficients, yet accurately predicts the perceptual attributes of Munsell samples under a selected standard lighting condition. We then added a feedback loop to adjust the model coefficients automatically, in response to switching between the cone fundamentals for different observers. This feature is intended to model long term contrast adaptation in the vision system by maintaining the average long‐term levels of color contrast. Incorporating this adaptation principle into color appearance models could allow better assessments of displays and illumination systems, to help improve color appearances for most observers.