Mathematically gifted students are precious human resources, educators of which make a great difference in helping them realize their potential. The retention of qualified teachers of mathematically gifted students is an issue worthy of in-depth exploration. In China, mathematics competitions are an important part of mathematics gifted education, and the teaching of the Mathematical Olympiad is a challenging profession with a high attrition rate. This qualitative study takes four seasoned and outstanding Chinese teachers as cases, collects data through individual semi-structured interviews, and uses the thematic analysis method based on Herzberg’s two-factor theory to analyze why they can persist in the field of Mathematical Olympiad teaching for more than 20 years. The results show that the motivation factors affecting retention are recognition, achievement, possibility of growth, work itself, and responsibility, and the hygiene factors are policy and interpersonal relationships. Motivation factors are the main reason for their long-term persistence, while hygiene factors are conducive to their persistence as a whole. Finally, enlightenment to educational policy and school management and suggestions for retention and development of teachers of the mathematically gifted are provided.