To explore the impact of academic burden on the physical and mental health of primary school students, combined with the results of the Programme for International Student Assessment (PISA) report in 2018, the relationship among the development of mathematical literacy, mathematics academic burden, and the physical and mental health of primary school students is studied. First, the relationship between mathematical literacy and mathematics anxiety is analyzed, and related influencing factors and measurement methods of mathematics anxiety are introduced. A questionnaire is then designed for primary school students’ mathematical stress, and the reliability and validity of the designed questionnaire are tested. Finally, a questionnaire survey is conducted on students, parents, and teachers in the third, fourth, and fifth grades of three standardized public primary schools. The results of the questionnaire survey show that students, teachers, and parents have a general understanding of the mathematics academic burden of primary school students at this stage. A total of 70% of teachers believe that primary school students have a heavy mathematics burden; 50% of parents think that primary school students are under heavy academic stress; 70% of primary school students believe that the heavy mathematics burden leads to reduced sleep time and extracurricular activities, which has a serious impact on the physical and mental health of primary school students. This research provides a reference for improving the current balance between education and students’ physical and mental health in China.