Burgeoning research has shown the pervasiveness of sound symbolism, a type of iconicity, in language. However, little work looks at how individual experiences and beliefs affect sound symbolic associations. We investigate pitch associations under the Frequency Code, which links high vs low pitch to small vs large body size and female vs male gender (via sexual dimorphism), long claimed to underlie “universal” meanings like submissiveness vs dominance. While such associations appear widespread, the Frequency Code assumes ideological links, e.g., between dominance and masculinity, which differ between individuals and cultures. In Implicit Association Task experiments with English-speaking listeners, we show high pitch is implicitly associated with small size and female gender, and low with large and male, following the Frequency Code. Crucially, though, the strength of these associations varies by social factors. Associations are stronger for male voices and listeners, particularly older men, as ideologies related to the Frequency Code (linking large size, strength, and dominance) tend to be stronger for men. The association of pitch with gender is stronger than with body size, reflecting stronger gender-based stereotypes. This work shows that social experience shapes and reinforces iconic associations, with important implications for how iconic meanings develop and are processed.