Color blindness (color vision deficiency) affects ~8% of males and ~ 0.4% females worldwide. Here we use the elicited lists method to investigate their semantic knowledge regarding “basic color terms” and their relationships. Lists were obtained from color vision deficient and normal observers. 32 dichromats (15 protanopes, 17 deuteranopes) and 32 normal trichromats (17 females, 15 males) diagnosed by a battery of color tests (Ishihara, City University Test, anomaloscope) wrote monolexemic lists of colors. Psychological salience of terms (ln(CSI)), adjacency between pairs of terms (ADJ; MDS) and the presence of clusters of terms defined on the basis of the Universals and evolution hypothesis were analyzed. All four groups of participants showed the same semantic memory structure: lists started with the cardinal primaries cluster (blue, red, yellow, green), followed by the achromatic primaries cluster (black and white), or the derived cluster (brown, orange, violet, pink, purple, and gray). After the clusters (cardinals, achromatics, and derived), a highly variable number of non‐basic terms appeared. This number was higher in normal trichromats. Non‐basic terms were not part of any cognitive cluster. The similarity in the lists of trichromats and dichromats suggest that both may acquire similar semantic knowledge about color terms. Several potential explanations are considered.