Sport and physical education are considered important platforms for introducing people from refugee backgrounds to their new environment. However, studies dealing with the specific role of physical education in this context are scarce, although it is often the first and sometimes only setting in which people from refugee backgrounds come into contact with sport in the host country. To this end, this study attempts to shed light on how teachers—considered key actors in the resettlement process—perceive students from refugee backgrounds and their integration in physical education. In this study, teachers’ perceptions are defined as individually constructed truths influenced by biographical circumstances (migration experiences), socio-demographic characteristics (gender) and the environment (school type). We used integration theories to operationalise teachers’ perceptions, from which we derived a three-dimensional structure: cultural, structural-positional and communicative-interactive dimensions. Drawing on this theoretical foundation, quantitative data from a questionnaire administered to 491 physical education teachers in Germany were analysed using exploratory factor analysis, t-tests and variance analysis. The factor analysis resulted in a three-factor solution. The mean value comparison of the factors on the descriptive level revealed that the communicative-interactive dimension had slightly higher values than the other two dimensions. Furthermore, there were significant differences in perceptions in terms of gender and school type. Based on our findings, we suggest that teachers should be made aware that their views on students are never neutral and are shaped by various factors, especially gender and school type.