This study aimed to determine psychological and physical differences in elementary and junior high school teachers during COVID-19. This questionnaire-based cross-sectional study was conducted among 427 teachers in Tokyo, Japan (between 15 and 30 October 2020). The questionnaire explored school type (elementary and middle schools), sex, age, and COVID-19 changes (psychological changes, physical changes, impact on work, and infection control issues perceived to be stressed). Post hoc tests for I cannot concentrate on work at all, found a significant difference for no change–improved and male teacher in elementary school female teacher in junior high school (p = 0.03). Regarding stress situation due to implementation of COVID-19 infection control, there were significant differences for disinfection work by teachers between male teachers in elementary school female teachers in junior high school (p = 0.04) and female teachers in elementary school female teachers in junior high school (p = 0.03). COVID-19 produced differences in psychological and physical changes between male and female teachers in elementary and junior high schools. Some experienced psychological and physical stress, whereas others showed improvement. Given that teachers’ mental health also affects students’ educational quality, it is important to understand and improve teachers’ psychological and physical circumstances and stress.