There are many factors and duties that novice teachers do not know about when they enter the profession. Isolation, for instance, affects physical education (PE) teachers because the position often comes with a secluded workplace adjacent to the gym and through the subject's marginalization. These challenges, among others, can send the novice teacher into shock, and there is a risk the teacher becomes disengaged from the profession. This study aimed to provide insights into PE teachers’ experiences of reality shock, isolation, and marginalization over time, as well as how they managed these challenges. Semi-structured interviews were conducted with nine Swedish PE teachers six to 10 years after graduation from PE teacher education (PETE). Qualitative content analysis was used. All participants retrospectively described a reality shock when they were novices, where non-subject-specific teacher assignments were experienced as more challenging compared to teaching PE. Further, isolation had a severe effect. However, the marginalization of PE was not that apparent as when they were novices. Over time, the participants created their own networks to mitigate, for example, isolation. In turn, marginalization seemed to be a remaining challenge to the participants despite an increased societal focus on health. Even though it is impossible for PETE to prepare teachers for every challenge they could face in the profession, PETE should inform preservice teachers of common phenomena such as isolation and marginalization, as well as how to create supporting networks. Further, we argue that perceived autonomy combined with systematic guidance could positively affect novice teachers’ development.