Even when people hold little prejudice themselves, expectations about how members of other groups perceive them may negatively influence interracial interactions. In four pre-registered experiments each using a full intergroup design with Black and White participants, we show that people infer negative meta-attitudes from out-group members’ whose appearance is phenotypically prototypical, which in turn leads to less favorable orientations towards intergroup contact, independent of personal attitudes. In Experiment 1, Black Americans but not White Americans, perceived phenotypically prototypical out-group members to hold less favorable meta-attitudes and this explained less favorable contact orientations. In Experiment 2, this pattern emerged for both groups of participants and was pronounced among stigma conscious individuals. Experiment 3 replicated and extended Experiment 2 with representative samples, further demonstrating that the effect of phenotypic prototypicality is pronounced among participants who report previous rejection by the out-group. In Experiment 4, direct evidence for the causal effect of the mediator meta-attitudes on orientations toward contact was obtained. In all studies, effects held controlling for participants’ general intergroup attitudes and experiences, demonstrating the unique role of attitudes at the meta-perceptual level in shaping intergroup relations. Participants also perceived phenotypically prototypical in-group members as having less favorable intergroup attitudes, suggesting a general tendency to infer meta-attitudes from phenotypic prototypicality. We discuss our results in light of previous research, highlight social implications, and suggest future directions.