Teaching physical education (PE) at a distance has been a major challenge for primary school teachers because younger students are not yet able to use Information Communication Technology (ICT) independently and their concentration is lower. In order to maintain the development of student's motor skills, it was very important to incorporate various PE games into all distance learning lessons. The aim of this study was to investigate whether primary school teachers conduct PE classes at all in the context of distance education and what constraints and challenges they encounter. The sample included 181 primary school teachers. For data collection, we used a questionnaire prepared and used for the analysis of teaching during the first declared epidemic in Slovenia in March 2020. The data were collected between January 15 and February 15, 2021. The results showed that teachers mostly taught the content of hiking, natural forms of movement, dance, and athletics. Dance was taught by more teachers teaching grades 1 to 3 (x2 = 4.731; g = 1; p = 0.030), and Theoretical Content was taught by more teachers teaching grades 4 to 6 (x2 = 19.434; g = 1; p < 0.001). The most common problem in distance education in PE was technological problems and students' lack of knowledge in using ICT. Compared to teachers with fewer years of service, teachers with the most years of service had the most difficulty providing assistive instructional technology (p = 0.001) and knowledge of using digital tools (p = 0.015; p = 0.31).