Despite inclusive education being high on the agenda of educational systems worldwide and part of international legislation and policy, particularly the 2030 Agenda for Sustainable Development, which emphasises inclusion and equality as the basis for quality education for all, there is still a need to identify how teachers in regular schools who teach students with special educational needs (SENs) experience their work. This study investigates the most significant stressors for primary school teachers teaching students with SENs. In addition, this study sets out to determine whether there were differences regarding stress between teachers based on their demographic characteristics and the type of SENs in the students they teach. The empirical study, based on a sample of 120 Slovenian teachers (89.2% females), showed that a lack of professional competency, classroom work, personal beliefs, and the burden of administrative work were the most significant perceived stressors when teaching students with SENs in regular primary school. Regarding students’ behaviour and classroom work, different types of SENs were also found to cause significant differences in stress perception. The study revealed that teaching experience and teacher training in SENs could be crucial stress reducers for primary school teachers who teach students with SENs.