Those who teach mathematically gifted high school students play a key role in both their identification and cultivation. Teachers who teach the Mathematical Olympiad in China work long hours and teach difficult content; they are under a significant amount of work-based pressure, and that is worthy of further study. This study analyzed the sources of stress for 33 Chinese teachers who teach the Mathematical Olympiad, collected data through semi-structured in-depth interviews, and adopted a subject analysis method based on Bronfenbrenner’s ecological system that considers the influences that the complexity of socio-cultural and environment have on individual emotions. It is divided into five structural or environmental systems in which human beings develop, namely the macroscopic, external, meso, micro, and chrono systems. The research results show that the greatest influences on these teachers’ stress come from the mesosystem and microsystem levels, and that the key players are students and school leaders. Educational policy and culture are found to be key factors from the macrosystem. Overall, long-term stress was seen to affect both teachers’ moods and their physical health. Finally, the results offered implications for education policy and school management and suggestions for the cultivation and management of mathematics teachers who teach the gifted. Limitations of the study are discussed, and directions for future research are proposed.