A longstanding and unresolved question is how observers construct a discrete set of color categories to partition and label the continuous variations in light spectra, and how these categories might reflect the neural representation of color. We explored the properties of color naming and its relationship to color appearance by analyzing individual differences in color-naming and hue-scaling patterns, using factor analysis of individual differences to identify separate and shared processes underlying hue naming (labeling) and hue scaling (color appearance). Observers labeled the hues of 36 stimuli spanning different angles in cone-opponent space, using a set of eight terms corresponding to primary (red, green, blue, yellow) or binary (orange, purple, blue-green, yellow-green) hues. The boundaries defining different terms varied mostly independently, reflecting the influence of at least seven to eight factors. This finding is inconsistent with conventional color-opponent models in which all colors derive from the relative responses of underlying red-green and blue-yellow dimensions. Instead, color categories may reflect qualitatively distinct attributes that are free to vary with the specific spectral stimuli they label. Inter-observer differences in color-naming were large and systematic, and we examined whether these differences were associated with differences in color appearance by comparing the hue-naming to color percepts assessed by hue scaling measured in the same observers (from Emery et al., submitted). Variability in both tasks again depended on multiple (7 or 8) factors, with some Varimax-rotated factors specific to hue naming or hue scaling, but others common to corresponding stimuli for both judgments. The latter suggests that at least some of the differences in how individuals name or categorize color are related to differences in how the stimuli are perceived.