The U.S. political landscape is increasingly polarized. Political polarization is made especially salient when visualized using a two-tone U.S. map with red Republican-leaning states and blue Democrat-leaning states. However, such dichotomization conceals the true spectrum of political opinion and voting patterns. We asked whether dichotomizing maps into red and blue states leads people to overestimate polarization compared to maps that represent voting patterns continuously using red to white to blue hue and saturation color gradients. And if so, is this polarizing effect due to partisan semantic associations with red and blue, or can any color mappings produce similar effects? Online U.S. participants estimated the hypothetical voting patterns of eight swing states depicted using dichotomous or continuous red/blue or orange/green color maps. The dichotomous red/blue maps produced more polarized predictions than the continuous red/blue maps, but there was no difference between dichotomous and continuous orange/green maps. This suggests the polarizing effect red/blue dichotomous maps have on predicted voting patterns can be mitigated by switching to novel colors. We conclude that the tribal associations and categorical nature of red/blue maps are not simply tools for illustrating political polarization in the U.S.—they may in fact be exacerbating them.