How do Swedes, who are not exposed to administrative routines of reporting race and ethnicity, perceive, and categorize faces with different phenotypical features? This study examines identity contestation that can occur and address how race affects the way you are perceived as Swedish. A sample of Swedish participants were asked to assign racial categories to images of faces with different phenotypes, identify the skin color of the faces and rate how ‘Swedish’ the faces are perceived. We also use eye‐tracking to explore whether participants look differently at the faces of different racial groups. The results show greater identity contestation among Latino and Black faces. Moreover, while Swedish respondents identify the skin color of faces self‐identified as Black as darker compared to other racial backgrounds, the differences in skin color reported between Asian, White and Latinos were very small. Despite these small differences in the perception of skin color among Asian, White, and Latino faces, faces self‐reported as White were rated as significantly more Swedish by the respondents compared Asian and Latino faces. All these results contribute to the understanding of not only how race matters in Sweden but also to the understanding of constructivist nature of race.