Invisible violence in school physical education can negatively affect students’ body and mind and is detrimental to students’ sports cognition, sports participation, and sound personality cultivation and shaping. We analyze the manifestation and causes of invisible violence in physical education and propose a path to eliminate it, so as to improve the quality of school physical education and promote students’ physical and mental health. In this study, 30 students were interviewed, relevant research data were collected, and coding and clustering were conducted using the rooting theory research method. The results of the study are as follows: Invisible violence in physical education is characterized by concealment, unconsciousness, and indirect harm. According to the 11 characteristics of invisible violence in physical education, invisible violence in physical education is classified as invisible behavioral violence, invisible verbal violence, and invisible discrimination violence. The root causes of invisible violence in physical education are physical education teachers’ lack of understanding of the value of physical education, misplaced transfer of bad emotions and teaching. The root causes of invisible violence in physical education are teachers’ lack of understanding of the value of physical education, misplaced emotions, and the wrong expression of teaching language. The three countermeasures to eliminate invisible violence in physical education are: reshaping physical education teachers’ professional beliefs, improving physical education teachers’ emotional self-management ability, and strengthening physical education teachers’ oral expression ability.